Building

Started building at 2022/07/02 18:54:35
Using pegged server, 1432 build
Calculating base
Updating mirror
Basing run on 7.2.0-1432 c44d315bd
Updating tree for run 02.07.2022-18.54
query is at f979279, changes since last good build: none
gometa is at 8625ed1, changes since last good build: 

ns_server is at c9545cf, changes since last good build: none
couchstore is at f0c688c, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at d419a87, changes since last good build: none
Switching indexing to unstable
indexing is at b49b9d4, changes since last good build: none
Switching plasma to unstable
plasma is at 85217ad, changes since last good build: 
fatal: Invalid revision range c1941310ea2784b8e6d5b6caaf6b8d70a0f461c6..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at a237ed7, changes since last good build: none
Switching testrunner to master
testrunner is at 6e16c8a, changes since last good build: none
Pulling in uncommitted change 175819 at refs/changes/19/175819/4
Total 5 (delta 2), reused 3 (delta 2)
[unstable 1a6d0494] MB-51825: kvsender - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 7 15:41:01 2022 +0530
 1 file changed, 42 insertions(+), 24 deletions(-)
Pulling in uncommitted change 176730 at refs/changes/30/176730/1
Total 5 (delta 4), reused 5 (delta 4)
[unstable 91b45146] MB-52151: Dont meter deletes in insert paths
 Author: Sai Krishna Teja Kommaraju 
 Date: Mon Jun 27 20:11:44 2022 +0530
 1 file changed, 11 insertions(+), 8 deletions(-)
Pulling in uncommitted change 176873 at refs/changes/73/176873/3
Total 3 (delta 1), reused 2 (delta 1)
[unstable d5c6581] MB-100: Fix shard addInstance for LSS cleanup
 Author: saptarshi.sen 
 Date: Wed Jun 29 11:07:35 2022 -0700
 1 file changed, 6 insertions(+)
Building cmakefiles and deps
Building main product

... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo


... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo

Testing

Started testing at 2022/07/02 19:26:34
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Jul 02 19:31:23 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 02 19:35:05 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 02 19:35:44 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Jul 02 19:37:04 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Jul 02 19:37:48 b" 'stop_on_failure': 'True'}"
Jul 02 19:37:48 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 02 19:37:48 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_1'}"
Jul 02 19:37:48 b'-->result: '
Jul 02 19:37:48 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Jul 02 19:37:48 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 02 19:37:48 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_2'}"
Jul 02 19:37:48 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:37:48 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Jul 02 19:37:48 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_3'}"
Jul 02 19:37:48 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:37:48 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:37:48 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Jul 02 19:37:48 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_4'}"
Jul 02 19:37:48 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:37:48 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:37:48 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 02 19:47:57 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Jul 02 19:48:32 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Jul 02 19:51:58 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Jul 02 19:55:34 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Jul 02 19:58:06 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Jul 02 19:58:06 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_5'}"
Jul 02 19:58:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Jul 02 19:58:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Jul 02 19:58:06 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_6'}"
Jul 02 19:58:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 02 19:58:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Jul 02 19:58:06 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_7'}"
Jul 02 19:58:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Jul 02 19:58:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Jul 02 19:58:06 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_19-26-56/test_8'}"
Jul 02 19:58:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 02 19:58:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 02 19:58:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Jul 02 19:58:06 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Jul 02 19:58:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 02 19:58:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 02 19:58:07 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Jul 02 19:58:07 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Jul 02 19:58:07 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Jul 02 19:58:07 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Jul 02 19:58:07 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Jul 02 19:58:07 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.01s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000000 items took 28.162238574s -> 568136.6542633991 items/s conflicts 35
--- PASS: TestInsertPerf (28.16s)
=== RUN   TestGetPerf
16000000 items took 5.776070447s -> 2.7700493175788997e+06 items/s
--- PASS: TestGetPerf (6.38s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3351,
"memory_used":            37360992,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749144,
"level1": 187776,
"level2": 47222,
"level3": 11903,
"level4": 2974,
"level5": 735,
"level6": 182,
"level7": 48,
"level8": 12,
"level9": 2,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [35911 108875 357987 462344 598654 671258 781160]
No of items in each range [35911 72964 249112 104357 136310 72604 109902 218840]
--- PASS: TestGetRangeSplitItems (0.60s)
=== RUN   TestBuilder
{
"node_count":             50000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3333,
"memory_used":            1866650928,
"node_allocs":            50000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37500395,
"level1": 9374070,
"level2": 2344403,
"level3": 586130,
"level4": 146617,
"level5": 36338,
"level6": 9033,
"level7": 2299,
"level8": 539,
"level9": 123,
"level10": 39,
"level11": 9,
"level12": 5,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 8.583943575s to build 50000000 items, 5.8248285e+06 items/sec
Took 419.70057ms to iterate 50000000 items
--- PASS: TestBuilder (9.00s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb/skiplist	44.397s
=== RUN   TestInsert
Set IO Concurrency: 716
--- PASS: TestInsert (0.01s)
=== RUN   TestInsertPerf
20000000 items took 52.289346284s -> 382487.09194744314 items/s snapshots_created 210 live_snapshots 1
--- PASS: TestInsertPerf (68.73s)
=== RUN   TestInsertDuplicates
--- PASS: TestInsertDuplicates (0.44s)
=== RUN   TestGetPerf
16000000 items took 11.924459459s -> 1.3417798982849475e+06 items/s
--- PASS: TestGetPerf (13.68s)
=== RUN   TestLoadStoreDisk
Inserting 1000000 items took 1.481884316s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       13,
"next_pointers_per_node": 1.3318,
"memory_used":            57308784,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750601,
"level1": 187435,
"level2": 46574,
"level3": 11594,
"level4": 2861,
"level5": 698,
"level6": 180,
"level7": 41,
"level8": 13,
"level9": 1,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
2022-07-02T20:00:25.330+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[65796 22435 54037 44787 47526 32086 26582 33884 24087 93557 28662 115319 56427 45855 124661 184299]] total count [1000000] snap count [1000018] size per shard [[526368 179480 432296 358296 380208 256688 212656 271072 192696 748456 229296 922552 451416 366840 997288 1474392]] total size [8000000] memoryInUse [57308784]
Storing to disk took 302.262196ms
Loading from disk took 295.294741ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3314,
"memory_used":            57303184,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750709,
"level1": 187328,
"level2": 46722,
"level3": 11482,
"level4": 2853,
"level5": 684,
"level6": 169,
"level7": 41,
"level8": 10,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestLoadStoreDisk (3.67s)
=== RUN   TestConcurrentLoadCloseFragmentation
Done Inserting 2000000 items
frag = NaN%
2022-07-02T20:00:31.160+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[12591 103139 71743 56690 54660 77615 79694 221942 113500 127930 179014 149733 50643 38924 47735 614447]] total count [2000000] snap count [2000000] size per shard [[100728 825112 573944 453520 437280 620920 637552 1775536 908000 1023440 1432112 1197864 405144 311392 381880 4915576]] total size [16000000] memoryInUse [114653228]
Done Storing to disk
frag = NaN%
Done Loading from disk
Done Closing
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentation (5.83s)
=== RUN   TestConcurrentLoadCloseFragmentationSmall
Done Inserting 10 items
frag = NaN%
2022-07-02T20:00:32.874+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationSmall (0.10s)
=== RUN   TestConcurrentLoadCloseFragmentationEmpty
Done Inserting 0 items
frag = NaN%
2022-07-02T20:00:32.966+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationEmpty (0.05s)
=== RUN   TestConcurrentCloseSingleNode
A
--- PASS: TestConcurrentCloseSingleNode (0.00s)
=== RUN   TestStoreDiskShutdown
Inserting 1000000 items took 1.420003524s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       16,
"next_pointers_per_node": 1.3332,
"memory_used":            57330688,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750329,
"level1": 186992,
"level2": 47016,
"level3": 11790,
"level4": 2903,
"level5": 727,
"level6": 188,
"level7": 45,
"level8": 7,
"level9": 2,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
Prepare returns error:  MemDB instance has been shutdown
--- PASS: TestStoreDiskShutdown (2.26s)
=== RUN   TestDelete
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.2000,
"memory_used":            572,
"node_allocs":            10,
"node_frees":             0,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.5000,
"memory_used":            620,
"node_allocs":            20,
"node_frees":             10,
"level_node_distribution":{
"level0": 5,
"level1": 5,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestDelete (1.00s)
=== RUN   TestGCPerf
final_node_count = 16000, average_live_node_count = 68371, wait_time_for_collection = 23ms
--- PASS: TestGCPerf (36.08s)
=== RUN   TestMemoryInUse
ItemsCount: 5000, MemoryInUse: 297120, NodesCount: 5000
ItemsCount: 0, MemoryInUse: 64, NodesCount: 0
--- PASS: TestMemoryInUse (1.01s)
=== RUN   TestFullScan
Inserting 1600000 items took 2.589716376s
{
"node_count":             1601466,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       14,
"next_pointers_per_node": 1.3331,
"memory_used":            91738208,
"node_allocs":            1600000,
"node_frees":             0,
"level_node_distribution":{
"level0": 1201665,
"level1": 299703,
"level2": 74976,
"level3": 18889,
"level4": 4619,
"level5": 1218,
"level6": 301,
"level7": 75,
"level8": 15,
"level9": 3,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Full iteration of 1600000 items took 334.124678ms
--- PASS: TestFullScan (4.23s)
=== RUN   TestVisitor
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3327,
"memory_used":            57322784,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750667,
"level1": 186806,
"level2": 46930,
"level3": 11706,
"level4": 2886,
"level5": 753,
"level6": 200,
"level7": 39,
"level8": 9,
"level9": 4,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 47.51184ms to iterate 1000000 items, 2.1047386e+07 items/s
shard - 0 count = 35666, range: 0-35665
shard - 1 count = 37931, range: 35666-73596
shard - 2 count = 26786, range: 73597-100382
shard - 3 count = 92250, range: 100383-192632
shard - 4 count = 39760, range: 192633-232392
shard - 5 count = 16119, range: 232393-248511
shard - 6 count = 40188, range: 248512-288699
shard - 7 count = 145644, range: 288700-434343
shard - 8 count = 54168, range: 434344-488511
shard - 9 count = 27543, range: 488512-516054
shard - 10 count = 8530, range: 516055-524584
shard - 11 count = 27457, range: 524585-552041
shard - 12 count = 61692, range: 552042-613733
shard - 13 count = 36472, range: 613734-650205
shard - 14 count = 32498, range: 650206-682703
shard - 15 count = 17646, range: 682704-700349
shard - 16 count = 34361, range: 700350-734710
shard - 17 count = 32848, range: 734711-767558
shard - 18 count = 33739, range: 767559-801297
shard - 19 count = 10121, range: 801298-811418
shard - 20 count = 99419, range: 811419-910837
shard - 21 count = 20239, range: 910838-931076
shard - 22 count = 14165, range: 931077-945241
shard - 23 count = 20852, range: 945242-966093
shard - 24 count = 9729, range: 966094-975822
shard - 25 count = 17126, range: 975823-992948
shard - 26 count = 7051, range: 992949-999999
shard - 27 count = 0, range: 0-0
shard - 28 count = 0, range: 0-0
shard - 29 count = 0, range: 0-0
shard - 30 count = 0, range: 0-0
shard - 31 count = 0, range: 0-0
--- PASS: TestVisitor (1.35s)
=== RUN   TestVisitorError
--- PASS: TestVisitorError (0.14s)
=== RUN   TestLoadDeltaStoreDisk
snap count = 1000000
2022-07-02T20:01:23.056+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 351587 194184 241793 44453 105483 21208 31171 0 0 0]] total count [989879] snap count [1000000] size per shard [[0 0 0 0 0 0 2812696 1553472 1934344 355624 843864 169664 249368 0 0 0]] total size [7919032] memoryInUse [116789888]
Storing to disk took 15.2972649s
Loading from disk took 935.757195ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3330,
"memory_used":            57327616,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750583,
"level1": 186840,
"level2": 46855,
"level3": 11764,
"level4": 2987,
"level5": 732,
"level6": 168,
"level7": 53,
"level8": 15,
"level9": 3,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Restored 10121
RestoredFailed 348271
--- PASS: TestLoadDeltaStoreDisk (17.96s)
=== RUN   TestExecuteConcurrGCWorkers
--- PASS: TestExecuteConcurrGCWorkers (0.63s)
=== RUN   TestCloseWithActiveIterators
--- PASS: TestCloseWithActiveIterators (12.55s)
=== RUN   TestDiskCorruption
Inserting 100000 items took 105.938587ms
{
"node_count":             99999,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       6,
"next_pointers_per_node": 1.3295,
"memory_used":            5727120,
"node_allocs":            100000,
"node_frees":             0,
"level_node_distribution":{
"level0": 75077,
"level1": 18862,
"level2": 4574,
"level3": 1124,
"level4": 274,
"level5": 67,
"level6": 15,
"level7": 6,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
2022-07-02T20:01:50.374+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[4497 11970 12117 1755 9056 13360 23660 6984 5329 6797 2262 2213 0 0 0 0]] total count [100000] snap count [100000] size per shard [[35976 95760 96936 14040 72448 106880 189280 55872 42632 54376 18096 17704 0 0 0 0]] total size [800000] memoryInUse [5727120]
Storing to disk took 87.596374ms
Loading from disk took 30.058233ms
--- PASS: TestDiskCorruption (0.29s)
=== RUN   TestSnapshotStats
--- PASS: TestSnapshotStats (0.00s)
=== RUN   TestInsertDeleteConcurrent
total items in snapshot: 8125871  items deleted: 1874129
{
"node_count":             8125871,
"soft_deletes":           0,
"read_conflicts":         9219,
"insert_conflicts":       135,
"next_pointers_per_node": 1.3340,
"memory_used":            465964348,
"node_allocs":            10000000,
"node_frees":             10000002,
"level_node_distribution":{
"level0": 6091747,
"level1": 1524463,
"level2": 382023,
"level3": 95885,
"level4": 23812,
"level5": 6002,
"level6": 1444,
"level7": 385,
"level8": 87,
"level9": 16,
"level10": 7,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
allocs: 20000002 frees: 20000002
--- PASS: TestInsertDeleteConcurrent (18.41s)
=== RUN   TestNodeList
--- PASS: TestNodeList (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb	188.710s
=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.035s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-07-02T20:02:13.479+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5598 us

2022-07-02T20:02:13.480+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-02T20:02:13.484+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-07-02T20:02:13.484+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-02T20:02:13.487+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 2m1.582254936s -> 82248.8446629315 items/s
Incr build: 10000000 items took 2m15.840547238s -> 73615.72228120855 items/s
Main Index: {
"node_count":             19858446,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       2,
"next_pointers_per_node": 1.3334,
"memory_used":            1871121142,
"node_allocs":            19858446,
"node_frees":             0,
"level_node_distribution":{
"level0": 14892954,
"level1": 3724591,
"level2": 930478,
"level3": 232723,
"level4": 58213,
"level5": 14713,
"level6": 3583,
"level7": 871,
"level8": 243,
"level9": 53,
"level10": 20,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (257.43s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	281.460s
=== RUN   TestConnPoolBasicSanity
2022-07-02T20:06:57.988+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-07-02T20:06:58.196+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:06:58.988+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-07-02T20:07:02.991+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-07-02T20:07:42.747+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:07:43.009+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.76s)
=== RUN   TestLongevity
2022-07-02T20:07:46.750+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-07-02T20:08:25.364+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:08:25.770+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.61s)
=== RUN   TestSustainedHighConns
2022-07-02T20:08:29.365+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Allocating 16 Connections
cp.curActConns = 0
Returning 3 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 4 Connections
cp.curActConns = 13
Returning 1 Connections
Allocating 12 Connections
cp.curActConns = 22
Returning 1 Connections
cp.curActConns = 23
Allocating 10 Connections
Returning 1 Connections
cp.curActConns = 32
Returning 3 Connections
Allocating 15 Connections
cp.curActConns = 33
Returning 4 Connections
cp.curActConns = 40
Returning 3 Connections
Allocating 8 Connections
cp.curActConns = 45
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 45
Returning 4 Connections
cp.curActConns = 42
Allocating 9 Connections
Returning 3 Connections
cp.curActConns = 48
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 57
Returning 4 Connections
cp.curActConns = 63
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 59
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 69
Returning 3 Connections
cp.curActConns = 69
Allocating 3 Connections
Returning 0 Connections
cp.curActConns = 72
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 81
Returning 0 Connections
Allocating 6 Connections
cp.curActConns = 81
Returning 1 Connections
cp.curActConns = 86
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 93
Returning 2 Connections
cp.curActConns = 92
Allocating 8 Connections
Returning 1 Connections
cp.curActConns = 99
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 97
Returning 2 Connections
Allocating 18 Connections
cp.curActConns = 105
Returning 2 Connections
cp.curActConns = 111
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 109
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 117
Returning 0 Connections
cp.curActConns = 127
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 127
Returning 2 Connections
Allocating 8 Connections
cp.curActConns = 129
Returning 1 Connections
cp.curActConns = 132
Returning 4 Connections
Allocating 8 Connections
cp.curActConns = 136
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 141
Returning 2 Connections
cp.curActConns = 147
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 153
Returning 1 Connections
cp.curActConns = 154
Allocating 7 Connections
Returning 0 Connections
cp.curActConns = 161
Returning 3 Connections
Allocating 2 Connections
cp.curActConns = 160
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 162
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 159
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 165
Returning 2 Connections
cp.curActConns = 171
Returning 2 Connections
Allocating 10 Connections
cp.curActConns = 177
Returning 0 Connections
cp.curActConns = 179
Returning 0 Connections
Allocating 15 Connections
cp.curActConns = 189
Returning 2 Connections
cp.curActConns = 192
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 192
Returning 1 Connections
Allocating 7 Connections
cp.curActConns = 198
Returning 4 Connections
Allocating 22 Connections
cp.curActConns = 201
Returning 2 Connections
cp.curActConns = 213
Returning 2 Connections
cp.curActConns = 212
Allocating 1 Connections
Returning 4 Connections
cp.curActConns = 209
Allocating 1 Connections
Returning 2 Connections
cp.curActConns = 208
Allocating 1 Connections
Returning 4 Connections
cp.curActConns = 205
Allocating 1 Connections
Returning 0 Connections
cp.curActConns = 206
Allocating 3 Connections
Returning 1 Connections
cp.curActConns = 208
Allocating 4 Connections
Returning 1 Connections
cp.curActConns = 211
Allocating 2 Connections
Returning 0 Connections
cp.curActConns = 213
Allocating 0 Connections
Returning 0 Connections
cp.curActConns = 213
Allocating 2 Connections
Returning 4 Connections
cp.curActConns = 211
Allocating 3 Connections
Returning 3 Connections
cp.curActConns = 211
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 208
Allocating 2 Connections
Returning 2 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 212
Allocating 1 Connections
Returning 1 Connections
cp.curActConns = 212
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 213
Allocating 4 Connections
Returning 4 Connections
cp.curActConns = 213
Allocating 0 Connections
Returning 3 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 209
Allocating 2 Connections
Returning 3 Connections
cp.curActConns = 208
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 207
Allocating 3 Connections
Returning 4 Connections
cp.curActConns = 206
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 207
Allocating 2 Connections
Returning 3 Connections
cp.curActConns = 206
Allocating 3 Connections
Returning 3 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 205
Allocating 0 Connections
Returning 0 Connections
cp.curActConns = 205
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 203
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 201
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 204
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 207
Allocating 0 Connections
Returning 4 Connections
cp.curActConns = 203
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 206
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 210
Allocating 2 Connections
Returning 0 Connections
cp.curActConns = 212
Allocating 0 Connections
Returning 2 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 206
Allocating 3 Connections
Returning 1 Connections
cp.curActConns = 208
Allocating 0 Connections
Returning 1 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 208
Allocating 4 Connections
Returning 2 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 211
Allocating 1 Connections
Returning 0 Connections
cp.curActConns = 212
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 210
Allocating 2 Connections
Returning 2 Connections
cp.curActConns = 210
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-07-02T20:09:24.442+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:09:25.393+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.08s)
=== RUN   TestLowWM
2022-07-02T20:09:28.444+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-07-02T20:10:28.461+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-02T20:11:28.478+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-02T20:11:33.943+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:11:34.480+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.50s)
=== RUN   TestTotalConns
2022-07-02T20:11:37.945+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-07-02T20:11:52.117+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:11:52.952+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.17s)
=== RUN   TestUpdateTickRate
2022-07-02T20:11:56.119+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-07-02T20:12:16.969+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-02T20:12:17.131+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	323.033s
Starting server: attempt 1

Functional tests

2022/07/02 20:14:29 In TestMain()
2022/07/02 20:14:29 otp node fetch error: json: cannot unmarshal string into Go value of type couchbase.Pool
2022/07/02 20:14:29 Initialising services with role: kv,n1ql on node: 127.0.0.1:9000
2022/07/02 20:14:30 Initialising web UI on node: 127.0.0.1:9000
2022/07/02 20:14:30 InitWebCreds, response is: {"newBaseUri":"http://127.0.0.1:9000/"}
2022/07/02 20:14:31 Setting data quota of 1500M and Index quota of 1500M
2022/07/02 20:14:32 Adding node: https://127.0.0.1:19001 with role: kv,index to the cluster
2022/07/02 20:14:38 AddNode: Successfully added node: 127.0.0.1:9001 (role kv,index), response: {"otpNode":"n_1@127.0.0.1"}
2022/07/02 20:14:43 Rebalance progress: 0
2022/07/02 20:14:48 Rebalance progress: 0
2022/07/02 20:14:53 Rebalance progress: 100
2022/07/02 20:14:58 Created bucket default, responseBody: 
2022/07/02 20:15:03 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022/07/02 20:15:03 Successfully initialised cluster
2022/07/02 20:15:03 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022-07-02T20:15:03.896+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = EOF. Kill Pipe.
2022-07-02T20:15:03.896+05:30 [Error] FollowerSyncProxy.receiveAndUpdateAcceptedEpoch(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2022-07-02T20:15:03.896+05:30 [Error] WatcherServer.runOnce() : Watcher fail to synchronized with peer 127.0.0.1:9106
2022/07/02 20:15:04 ChangeIndexerSettings: Host  Port 0 Nodes []
2022/07/02 20:15:04 Changing config key queryport.client.settings.backfillLimit to value 0
2022/07/02 20:15:04 Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused


goroutine 1 [running]:
panic({0xe690e0, 0xc00050ee20})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:941 +0x397 fp=0xc0000d3ca8 sp=0xc0000d3be8 pc=0x43a6b7
log.Panicf({0x1001b54?, 0x1e?}, {0xc0000d3df8?, 0x27?, 0xe5e7a0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/log/log.go:392 +0x67 fp=0xc0000d3cf0 sp=0xc0000d3ca8 pc=0x5bc127
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/functionaltests.TestMain(0x445111?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:104 +0x5c7 fp=0xc0000d3ec8 sp=0xc0000d3cf0 pc=0xd167a7
main.main()
	_testmain.go:479 +0x1d3 fp=0xc0000d3f80 sp=0xc0000d3ec8 pc=0xdec093
runtime.main()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:250 +0x212 fp=0xc0000d3fe0 sp=0xc0000d3f80 pc=0x43d232
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000d3fe8 sp=0xc0000d3fe0 pc=0x46db21

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cfb0 sp=0xc00005cf90 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.forcegchelper()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:301 +0xad fp=0xc00005cfe0 sp=0xc00005cfb0 pc=0x43d48d
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46db21
created by runtime.init.6
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:289 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005d790 sp=0xc00005d770 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005d7c8 sp=0xc00005d790 pc=0x428757
runtime.gcenable.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0x41e306
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x46db21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:177 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x23234b0cbba6?, 0x10000?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005df20 sp=0xc00005df00 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005dfc8 sp=0xc00005df20 pc=0x426565
runtime.gcenable.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005dfe0 sp=0xc00005dfc8 pc=0x41e2a6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x46db21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:178 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0x1079cd8?, 0x0?, 0x20?, 0x2000000020?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c630 sp=0xc00005c610 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.runfinq()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc00005c7e0 sp=0xc00005c630 pc=0x41d3b3
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46db21
created by runtime.createfing
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mfinal.go:157 +0x45

goroutine 18 [select]:
runtime.gopark(0xc00006af98?, 0x2?, 0xa7?, 0xd6?, 0xc00006af8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006ae18 sp=0xc00006adf8 pc=0x43d5f6
runtime.selectgo(0xc00006af98, 0xc00006af88, 0x40af19?, 0x0, 0x8?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00006af58 sp=0xc00006ae18 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc00013a0d8)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:394 +0x67 fp=0xc00006afc8 sp=0xc00006af58 pc=0x783d87
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:556 +0x26 fp=0xc00006afe0 sp=0xc00006afc8 pc=0x7849e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46db21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:556 +0x37a

goroutine 19 [select]:
runtime.gopark(0xc000058798?, 0x2?, 0x0?, 0x0?, 0xc00005878c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000058608 sp=0xc0000585e8 pc=0x43d5f6
runtime.selectgo(0xc000058798, 0xc000058788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000058748 sp=0xc000058608 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc00013a0f0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:314 +0x85 fp=0xc0000587c8 sp=0xc000058748 pc=0x7837a5
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:557 +0x26 fp=0xc0000587e0 sp=0xc0000587c8 pc=0x784986
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x46db21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:557 +0x3ca

goroutine 20 [IO wait]:
runtime.gopark(0xc000102680?, 0xc00004ea00?, 0x70?, 0x98?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000549800 sp=0xc0005497e0 pc=0x43d5f6
runtime.netpollblock(0xc000189000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000549838 sp=0xc000549800 pc=0x436097
internal/poll.runtime_pollWait(0x7f63d8ece0d8, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000549858 sp=0xc000549838 pc=0x468109
internal/poll.(*pollDesc).wait(0xc000032200?, 0xc000189000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000549880 sp=0xc000549858 pc=0x4a1032
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000032200, {0xc000189000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000549900 sp=0xc000549880 pc=0x4a239a
net.(*netFD).Read(0xc000032200, {0xc000189000?, 0xc000072f50?, 0x550c3d?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc000549948 sp=0xc000549900 pc=0x667349
net.(*conn).Read(0xc000010500, {0xc000189000?, 0x11?, 0xc000549a68?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc000549990 sp=0xc000549948 pc=0x6775c5
bufio.(*Reader).Read(0xc000088540, {0xc000176001, 0x5ff, 0x452834?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc0005499c8 sp=0xc000549990 pc=0x51f4d4
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x191?, {0xc000176001?, 0x8?, 0x20?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc0005499f8 sp=0xc0005499c8 pc=0x7d3e05
encoding/json.(*Decoder).refill(0xc000133400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000549a48 sp=0xc0005499f8 pc=0x563d1f
encoding/json.(*Decoder).readValue(0xc000133400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000549a98 sp=0xc000549a48 pc=0x56391b
encoding/json.(*Decoder).Decode(0xc000133400, {0xe99a60, 0xc00010e3e0})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000549ac8 sp=0xc000549a98 pc=0x563578
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc00010e3c0, 0xc000118480)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc000549b08 sp=0xc000549ac8 pc=0x7d3065
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc000126690?, 0x4cc288?)
	:1 +0x2a fp=0xc000549b28 sp=0xc000549b08 pc=0x7d5b8a
net/rpc.(*Server).readRequestHeader(0xc000126690, {0x118c7d8, 0xc000112690})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:587 +0x66 fp=0xc000549bf8 sp=0xc000549b28 pc=0x7d2786
net/rpc.(*Server).readRequest(0x0?, {0x118c7d8, 0xc000112690})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:547 +0x3b fp=0xc000549cd0 sp=0xc000549bf8 pc=0x7d22db
net/rpc.(*Server).ServeCodec(0xc000126690, {0x118c7d8?, 0xc000112690})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:462 +0x87 fp=0xc000549dc8 sp=0xc000549cd0 pc=0x7d1a07
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000058f60?, 0xc00006ffa0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc000549f38 sp=0xc000549dc8 pc=0x7d4559
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0x1183100?, 0xc00000e660?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc000549f70 sp=0xc000549f38 pc=0x7d4c58
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc000124270)
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc000549fc0 sp=0xc000549f70 pc=0x7de45d
github.com/couchbase/cbauth.startDefault.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc000549fe0 sp=0xc000549fc0 pc=0x7de745
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x46db21
created by github.com/couchbase/cbauth.startDefault
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9

goroutine 50 [GC worker (idle)]:
runtime.gopark(0xc00005e798?, 0xc00005e788?, 0x19?, 0xaf?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005e758 sp=0xc00005e738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005e7e0 sp=0xc00005e758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 32 [GC worker (idle)]:
runtime.gopark(0xc0004b89c0?, 0xc000059fd0?, 0x31?, 0x59?, 0xc0004c4380?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000059f58 sp=0xc000059f38 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000059fe0 sp=0xc000059f58 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 36 [select]:
runtime.gopark(0xc0004d4f68?, 0x4?, 0x3?, 0x0?, 0xc0004d4db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004d4c00 sp=0xc0004d4be0 pc=0x43d5f6
runtime.selectgo(0xc0004d4f68, 0xc0004d4da8, 0xc00011bbc0?, 0x0, 0x1?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004d4d40 sp=0xc0004d4c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc000542000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc0004d4fc8 sp=0xc0004d4d40 pc=0x76a045
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc0004d4fe0 sp=0xc0004d4fc8 pc=0x767c26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004d4fe8 sp=0xc0004d4fe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 66 [IO wait]:
runtime.gopark(0xc0004cc340?, 0xc00004c500?, 0x10?, 0xba?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00022b9a0 sp=0xc00022b980 pc=0x43d5f6
runtime.netpollblock(0xc000510000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00022b9d8 sp=0xc00022b9a0 pc=0x436097
internal/poll.runtime_pollWait(0x7f63d8ecdd18, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00022b9f8 sp=0xc00022b9d8 pc=0x468109
internal/poll.(*pollDesc).wait(0xc000032b80?, 0xc000510000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00022ba20 sp=0xc00022b9f8 pc=0x4a1032
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000032b80, {0xc000510000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00022baa0 sp=0xc00022ba20 pc=0x4a239a
net.(*netFD).Read(0xc000032b80, {0xc000510000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc00022bae8 sp=0xc00022baa0 pc=0x667349
net.(*conn).Read(0xc000010590, {0xc000510000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc00022bb30 sp=0xc00022bae8 pc=0x6775c5
net/http.(*persistConn).Read(0xc00025a360, {0xc000510000?, 0x1000?, 0x1000?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1929 +0x4e fp=0xc00022bb90 sp=0xc00022bb30 pc=0x76862e
bufio.(*Reader).fill(0xc00020aea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc00022bbc8 sp=0xc00022bb90 pc=0x51ef03
bufio.(*Reader).ReadSlice(0xc00020aea0, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc00022bc18 sp=0xc00022bbc8 pc=0x51faef
net/http/internal.readChunkLine(0x400?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc00022bc68 sp=0xc00022bc18 pc=0x706565
net/http/internal.(*chunkedReader).beginChunk(0xc00009cff0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc00022bc98 sp=0xc00022bc68 pc=0x705fe8
net/http/internal.(*chunkedReader).Read(0xc00009cff0, {0xc0005d6000?, 0x5?, 0xc000540568?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc00022bd18 sp=0xc00022bc98 pc=0x7062ae
net/http.(*body).readLocked(0xc0005ce000, {0xc0005d6000?, 0x7f63d868d6b8?, 0x7f63d9b7ff35?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:844 +0x3c fp=0xc00022bd68 sp=0xc00022bd18 pc=0x75d25c
net/http.(*body).Read(0x1010000000000?, {0xc0005d6000?, 0x0?, 0x7f64004b15b8?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:836 +0x125 fp=0xc00022bde0 sp=0xc00022bd68 pc=0x75d125
net/http.(*bodyEOFSignal).Read(0xc0005ce040, {0xc0005d6000, 0x200, 0x200})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2774 +0x142 fp=0xc00022be60 sp=0xc00022bde0 pc=0x76cd62
encoding/json.(*Decoder).refill(0xc000132000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:165 +0x17f fp=0xc00022beb0 sp=0xc00022be60 pc=0x563d1f
encoding/json.(*Decoder).readValue(0xc000132000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:140 +0xbb fp=0xc00022bf00 sp=0xc00022beb0 pc=0x56391b
encoding/json.(*Decoder).Decode(0xc000132000, {0xe3ebc0, 0xc0005d4000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:63 +0x78 fp=0xc00022bf30 sp=0xc00022bf00 pc=0x563578
github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x10b fp=0xc00022bfe0 sp=0xc00022bf30 pc=0x9c192b
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00022bfe8 sp=0xc00022bfe0 pc=0x46db21
created by github.com/couchbase/cbauth/metakv.doRunObserveChildren
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:280 +0x2eb

goroutine 37 [select]:
runtime.gopark(0xc00006ff90?, 0x2?, 0xd8?, 0xfd?, 0xc00006ff24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006fd90 sp=0xc00006fd70 pc=0x43d5f6
runtime.selectgo(0xc00006ff90, 0xc00006ff20, 0xc000514180?, 0x0, 0xc0004cb0b0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00006fed0 sp=0xc00006fd90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc000542000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00006ffc8 sp=0xc00006fed0 pc=0x76b135
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x767bc6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 12 [select]:
runtime.gopark(0xc0004d0f68?, 0x4?, 0x3?, 0x0?, 0xc0004d0db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004d0c00 sp=0xc0004d0be0 pc=0x43d5f6
runtime.selectgo(0xc0004d0f68, 0xc0004d0da8, 0xc000504540?, 0x0, 0xc0000b2101?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004d0d40 sp=0xc0004d0c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc00014b440)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc0004d0fc8 sp=0xc0004d0d40 pc=0x76a045
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc0004d0fe0 sp=0xc0004d0fc8 pc=0x767c26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004d0fe8 sp=0xc0004d0fe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 33 [GC worker (idle)]:
runtime.gopark(0x232a9e763fb3?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005a758 sp=0xc00005a738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005a7e0 sp=0xc00005a758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x232a9e775ef0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053a758 sp=0xc00053a738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00053a7e0 sp=0xc00053a758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053a7e8 sp=0xc00053a7e0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 13 [select]:
runtime.gopark(0xc0004d5f90?, 0x2?, 0xd8?, 0x5d?, 0xc0004d5f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004d5d90 sp=0xc0004d5d70 pc=0x43d5f6
runtime.selectgo(0xc0004d5f90, 0xc0004d5f20, 0xc00011bd80?, 0x0, 0xc000506900?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004d5ed0 sp=0xc0004d5d90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc00014b440)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc0004d5fc8 sp=0xc0004d5ed0 pc=0x76b135
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc0004d5fe0 sp=0xc0004d5fc8 pc=0x767bc6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004d5fe8 sp=0xc0004d5fe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 54 [chan receive]:
runtime.gopark(0x3?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000227e78 sp=0xc000227e58 pc=0x43d5f6
runtime.chanrecv(0xc00020a240, 0xc000227f90, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:577 +0x56c fp=0xc000227f08 sp=0xc000227e78 pc=0x40b5ac
runtime.chanrecv2(0xc0002062b0?, 0x16?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:445 +0x18 fp=0xc000227f30 sp=0xc000227f08 pc=0x40b018
github.com/couchbase/goutils/systemeventlog.(*SystemEventLoggerImpl).logEvents(0xc00012e070)
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:186 +0xb7 fp=0xc000227fc8 sp=0xc000227f30 pc=0xb72697
github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger.func1()
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x26 fp=0xc000227fe0 sp=0xc000227fc8 pc=0xb71fa6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000227fe8 sp=0xc000227fe0 pc=0x46db21
created by github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x1d6

goroutine 57 [select]:
runtime.gopark(0xc00053f798?, 0x2?, 0x8?, 0x0?, 0xc00053f78c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053f618 sp=0xc00053f5f8 pc=0x43d5f6
runtime.selectgo(0xc00053f798, 0xc00053f788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00053f758 sp=0xc00053f618 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*schedTokenMonitor).updater(0xc000126550)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2389 +0x92 fp=0xc00053f7c8 sp=0xc00053f758 pc=0xcb5e32
github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x26 fp=0xc00053f7e0 sp=0xc00053f7c8 pc=0xcb4766
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053f7e8 sp=0xc00053f7e0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x2e5

goroutine 58 [select]:
runtime.gopark(0xc000197ba0?, 0x3?, 0x0?, 0x30?, 0xc000197b1a?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000197978 sp=0xc000197958 pc=0x43d5f6
runtime.selectgo(0xc000197ba0, 0xc000197b14, 0x3?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000197ab8 sp=0xc000197978 pc=0x44d072
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc000112d30?, {0x101729d, 0x1b}, 0xc000197e68, 0xc000042de0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:301 +0x429 fp=0xc000197e40 sp=0xc000197ab8 pc=0x9c1489
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:259
github.com/couchbase/cbauth/metakv.RunObserveChildren({0x101729d?, 0x0?}, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x58 fp=0xc000197e88 sp=0xc000197e40 pc=0x9c1a38
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0?, {0x0?, 0x0?})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1576 +0xc7 fp=0xc000197f00 sp=0xc000197e88 pc=0xb77627
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc000197fa0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc000197f38 sp=0xc000197f00 pc=0xad7aa3
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1581 +0xdf fp=0xc000197fe0 sp=0xc000197f38 pc=0xb774df
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000197fe8 sp=0xc000197fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1569 +0xaf

goroutine 83 [select]:
runtime.gopark(0xc000227750?, 0x2?, 0x0?, 0x30?, 0xc000227714?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000227590 sp=0xc000227570 pc=0x43d5f6
runtime.selectgo(0xc000227750, 0xc000227710, 0x103f22e?, 0x0, 0xe685e0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0002276d0 sp=0xc000227590 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).logstats(0xc000032480, 0xc00021e2a0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1844 +0x238 fp=0xc0002277c0 sp=0xc0002276d0 pc=0xca27d8
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1819 +0x2a fp=0xc0002277e0 sp=0xc0002277c0 pc=0xca244a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002277e8 sp=0xc0002277e0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1819 +0x2ed

goroutine 61 [select]:
runtime.gopark(0xc00022ef68?, 0x4?, 0x48?, 0xec?, 0xc00022edb0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00022ec00 sp=0xc00022ebe0 pc=0x43d5f6
runtime.selectgo(0xc00022ef68, 0xc00022eda8, 0xc0005ce000?, 0x0, 0xc00053fd01?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00022ed40 sp=0xc00022ec00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc00025a360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc00022efc8 sp=0xc00022ed40 pc=0x76a045
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc00022efe0 sp=0xc00022efc8 pc=0x767c26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00022efe8 sp=0xc00022efe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 62 [select]:
runtime.gopark(0xc00022af90?, 0x2?, 0xd8?, 0xad?, 0xc00022af24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00022ad90 sp=0xc00022ad70 pc=0x43d5f6
runtime.selectgo(0xc00022af90, 0xc00022af20, 0xc000504100?, 0x0, 0xc000203ad0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00022aed0 sp=0xc00022ad90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc00025a360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00022afc8 sp=0xc00022aed0 pc=0x76b135
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc00022afe0 sp=0xc00022afc8 pc=0x767bc6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00022afe8 sp=0xc00022afe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 67 [select]:
runtime.gopark(0xc000547f38?, 0x2?, 0x0?, 0x30?, 0xc000547f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000547d58 sp=0xc000547d38 pc=0x43d5f6
runtime.selectgo(0xc000547f38, 0xc000547f20, 0xc0005dc150?, 0x0, 0xc00003b1d0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000547e98 sp=0xc000547d58 pc=0x44d072
github.com/couchbase/gometa/protocol.RunWatcherServerWithRequest2({0xc00003a9b0, 0xe}, {0x1185af8, 0xc0005da240}, {0x1192298, 0xc0005da240}, {0x1191780, 0x1a5b0f8}, 0xc0005dc0e0, 0xc0000428a0, ...)
	/opt/build/goproj/src/github.com/couchbase/gometa/protocol/watcherServer.go:72 +0x20e fp=0xc000547f68 sp=0xc000547e98 pc=0xc1838e
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4082 +0x9f fp=0xc000547fe0 sp=0xc000547f68 pc=0xc7723f
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4082 +0x1f8

goroutine 63 [select]:
runtime.gopark(0xc00053beb8?, 0x2?, 0xc0?, 0x3?, 0xc00053be90?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053bd18 sp=0xc00053bcf8 pc=0x43d5f6
runtime.selectgo(0xc00053beb8, 0xc00053be8c, 0x412065?, 0x0, 0x1031c9e?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00053be58 sp=0xc00053bd18 pc=0x44d072
github.com/couchbase/indexing/secondary/manager/client.(*watcher).waitForReady(0xc0005da240, 0xc0000428a0, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:5749 +0x239 fp=0xc00053bf20 sp=0xc00053be58 pc=0xc82159
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).retryHelper(0xc0002324d0, 0x0?, 0x0?, {0xc00003a9b0, 0xe}, {0xc00062e4e0, 0x1b}, 0x0?, 0xc0004b89e0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4016 +0x65 fp=0xc00053bf88 sp=0xc00053bf20 pc=0xc76a45
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:337 +0x46 fp=0xc00053bfe0 sp=0xc00053bf88 pc=0xc600a6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:337 +0x4d4

goroutine 82 [select]:
runtime.gopark(0xc000226f90?, 0x2?, 0x0?, 0x0?, 0xc000226f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000226e18 sp=0xc000226df8 pc=0x43d5f6
runtime.selectgo(0xc000226f90, 0xc000226f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000226f58 sp=0xc000226e18 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).listenMetaChange(0xc000032480, 0xc00021e2a0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1825 +0x70 fp=0xc000226fc0 sp=0xc000226f58 pc=0xca2550
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1818 +0x2a fp=0xc000226fe0 sp=0xc000226fc0 pc=0xca24aa
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000226fe8 sp=0xc000226fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1818 +0x28a

goroutine 64 [runnable]:
runtime.gopark(0xc000194ea8?, 0x6?, 0x68?, 0x4b?, 0xc000194cbc?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000194b20 sp=0xc000194b00 pc=0x43d5f6
runtime.selectgo(0xc000194ea8, 0xc000194cb0, 0x1008ec2?, 0x0, 0xc000194c98?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000194c60 sp=0xc000194b20 pc=0x44d072
net/http.(*persistConn).roundTrip(0xc000542000, 0xc00011bb80)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2620 +0x974 fp=0xc000194f18 sp=0xc000194c60 pc=0x76bff4
net/http.(*Transport).roundTrip(0x19a3920, 0xc000122e00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:594 +0x7c9 fp=0xc000195150 sp=0xc000194f18 pc=0x75fae9
net/http.(*Transport).RoundTrip(0xc000122e00?, 0x1182880?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc000195170 sp=0xc000195150 pc=0x747d99
net/http.send(0xc000122d00, {0x1182880, 0x19a3920}, {0xf8e640?, 0x48e801?, 0x1a24aa0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:252 +0x5d8 fp=0xc000195350 sp=0xc000195170 pc=0x7096b8
net/http.(*Client).send(0xc0004caf90, 0xc000122d00, {0x7f64004bad50?, 0x150?, 0x1a24aa0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:176 +0x9b fp=0xc0001953c8 sp=0xc000195350 pc=0x708f5b
net/http.(*Client).do(0xc0004caf90, 0xc000122d00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:725 +0x8f5 fp=0xc0001955c8 sp=0xc0001953c8 pc=0x70b395
net/http.(*Client).Do(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc000116810?, 0x1b?}, 0xc000195960, {0x0, 0x0}, 0x0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0001956c8 sp=0xc0001955c8 pc=0x7e3749
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc0002150e0, {0x10010b8, 0x6}, {0xeec5c0?, 0xc00003a001?}, {0xe40380, 0xc00011e798}, 0xc000042960?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc000195910 sp=0xc0001956c8 pc=0x806033
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth({0xc000042960, 0x3f}, {0x1182200?, 0xc000478c80})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:594 +0x125 fp=0xc000195988 sp=0xc000195910 pc=0x807fe5
github.com/couchbase/indexing/secondary/dcp.Connect({0xc000042960, 0x3f})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:600 +0xce fp=0xc000195b20 sp=0xc000195988 pc=0x80818e
github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier({0xc000042960, 0x3f}, {0x1002174, 0x7}, {0x1005cca, 0xb})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:239 +0x1a8 fp=0xc000195da0 sp=0xc000195b20 pc=0xada3a8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).watchClusterChanges(0xc000232420)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2071 +0xff fp=0xc000195fc8 sp=0xc000195da0 pc=0xcb3b3f
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x26 fp=0xc000195fe0 sp=0xc000195fc8 pc=0xca72c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000195fe8 sp=0xc000195fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x5c5

goroutine 65 [chan receive]:
runtime.gopark(0xc000127728?, 0xc000213d40?, 0x20?, 0xaf?, 0x459d45?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053aec8 sp=0xc00053aea8 pc=0x43d5f6
runtime.chanrecv(0xc00020ab40, 0x0, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:577 +0x56c fp=0xc00053af58 sp=0xc00053aec8 pc=0x40b5ac
runtime.chanrecv1(0xdf8475800?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:440 +0x18 fp=0xc00053af80 sp=0xc00053af58 pc=0x40afd8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).logstats(0xc000232420)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:1410 +0x79 fp=0xc00053afc8 sp=0xc00053af80 pc=0xcae8d9
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x26 fp=0xc00053afe0 sp=0xc00053afc8 pc=0xca7266
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053afe8 sp=0xc00053afe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x605

goroutine 45 [IO wait]:
runtime.gopark(0xc0004cc9c0?, 0xc000050f00?, 0x60?, 0xb7?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00019b6f0 sp=0xc00019b6d0 pc=0x43d5f6
runtime.netpollblock(0xc0001e3000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00019b728 sp=0xc00019b6f0 pc=0x436097
internal/poll.runtime_pollWait(0x7f63d8ecde08, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00019b748 sp=0xc00019b728 pc=0x468109
internal/poll.(*pollDesc).wait(0xc00011e400?, 0xc0001e3000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00019b770 sp=0xc00019b748 pc=0x4a1032
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00011e400, {0xc0001e3000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00019b7f0 sp=0xc00019b770 pc=0x4a239a
net.(*netFD).Read(0xc00011e400, {0xc0001e3000?, 0xc00019b8b8?, 0x551925?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc00019b838 sp=0xc00019b7f0 pc=0x667349
net.(*conn).Read(0xc00051e380, {0xc0001e3000?, 0x198?, 0xe54fc0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc00019b880 sp=0xc00019b838 pc=0x6775c5
net/http.(*persistConn).Read(0xc00014b440, {0xc0001e3000?, 0xc00019b910?, 0xc0005562e0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1929 +0x4e fp=0xc00019b8e0 sp=0xc00019b880 pc=0x76862e
bufio.(*Reader).fill(0xc00051ac00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc00019b918 sp=0xc00019b8e0 pc=0x51ef03
bufio.(*Reader).ReadSlice(0xc00051ac00, 0x5b?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc00019b968 sp=0xc00019b918 pc=0x51faef
net/http/internal.readChunkLine(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc00019b9b8 sp=0xc00019b968 pc=0x706565
net/http/internal.(*chunkedReader).beginChunk(0xc000202c30)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc00019b9e8 sp=0xc00019b9b8 pc=0x705fe8
net/http/internal.(*chunkedReader).Read(0xc000202c30, {0xc000634000?, 0x0?, 0xf?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc00019ba68 sp=0xc00019b9e8 pc=0x7062ae
net/http.(*body).readLocked(0xc000504540, {0xc000634000?, 0x0?, 0x1?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:844 +0x3c fp=0xc00019bab8 sp=0xc00019ba68 pc=0x75d25c
net/http.(*body).Read(0xc0004e7622?, {0xc000634000?, 0xc0004cb5c0?, 0xc0000e0d80?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:836 +0x125 fp=0xc00019bb30 sp=0xc00019bab8 pc=0x75d125
net/http.(*bodyEOFSignal).Read(0xc000504580, {0xc000634000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2774 +0x142 fp=0xc00019bbb0 sp=0xc00019bb30 pc=0x76cd62
bufio.(*Reader).fill(0xc00019bf58)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc00019bbe8 sp=0xc00019bbb0 pc=0x51ef03
bufio.(*Reader).ReadSlice(0xc00019bf58, 0x65?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc00019bc38 sp=0xc00019bbe8 pc=0x51faef
bufio.(*Reader).collectFragments(0xf6de20?, 0x28?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:446 +0x74 fp=0xc00019bce8 sp=0xc00019bc38 pc=0x51ff34
bufio.(*Reader).ReadBytes(0xc00020aa80?, 0x5d?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:474 +0x1d fp=0xc00019bd68 sp=0xc00019bce8 pc=0x5200dd
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).waitForChange(0xc00050a3f0?, 0x2c?, 0x1030f37?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:620 +0x4b fp=0xc00019be18 sp=0xc00019bd68 pc=0xad45cb
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).notifier(0xc0000f00e0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:709 +0x493 fp=0xc00019bfc8 sp=0xc00019be18 pc=0xad4e73
github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x26 fp=0xc00019bfe0 sp=0xc00019bfc8 pc=0xad07e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00019bfe8 sp=0xc00019bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x336
signal: aborted (core dumped)
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests	35.921s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 53120    0 53120    0     0  6456k      0 --:--:-- --:--:-- --:--:-- 7410k
2022/07/02 20:15:07 In TestMain()
2022/07/02 20:15:07 Changing config key indexer.api.enableTestServer to value true
2022/07/02 20:15:07 Using memory_optimized for creating indexes
2022/07/02 20:15:07 Changing config key indexer.settings.storage_mode to value memory_optimized
=== RUN   TestRangeWithConcurrentAddMuts
2022/07/02 20:15:12 In TestRangeWithConcurrentAddMuts()
2022/07/02 20:15:12 In DropAllSecondaryIndexes()
2022/07/02 20:15:12 Generating JSON docs
2022/07/02 20:15:12 Setting initial JSON docs in KV
2022/07/02 20:15:13 All indexers are active
2022/07/02 20:15:13 Creating a 2i
2022/07/02 20:15:16 Created the secondary index index_company. Waiting for it become active
2022/07/02 20:15:16 Index is 11093852903095607939 now active
2022/07/02 20:15:16 In Range Scan for Thread 1: 
2022/07/02 20:15:16 CreateDocs:: Creating mutations
2022/07/02 20:15:16 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
--- PASS: TestRangeWithConcurrentAddMuts (123.77s)
=== RUN   TestRangeWithConcurrentDelMuts
2022/07/02 20:17:16 In TestRangeWithConcurrentDelMuts()
2022/07/02 20:17:16 Generating JSON docs
2022-07-02T20:17:16.248+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:39740->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/07/02 20:17:17 Setting initial JSON docs in KV
2022/07/02 20:17:38 All indexers are active
2022/07/02 20:17:38 Creating a 2i
2022/07/02 20:17:38 Index found:  index_company
2022/07/02 20:17:38 In Range Scan for Thread 1: 
2022/07/02 20:17:38 CreateDocs:: Delete mutations
2022/07/02 20:17:38 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
--- PASS: TestRangeWithConcurrentDelMuts (142.88s)
2022-07-02T20:19:39.131+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:43892->127.0.0.1:9106: use of closed network connection. Kill Pipe.
=== RUN   TestScanWithConcurrentIndexOps
2022/07/02 20:19:39 In TestScanWithConcurrentIndexOps()
2022/07/02 20:19:39 Generating JSON docs
2022/07/02 20:19:44 Setting initial JSON docs in KV
2022/07/02 20:20:55 All indexers are active
2022/07/02 20:20:55 Creating a 2i
2022/07/02 20:20:55 Index found:  index_company
2022/07/02 20:20:55 In Range Scan for Thread 1: 
2022/07/02 20:20:55 Create and Drop index operations
2022/07/02 20:20:55 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022/07/02 20:21:12 Created the secondary index index_age. Waiting for it become active
2022/07/02 20:21:12 Index is 585598662685292285 now active
2022/07/02 20:21:31 Created the secondary index index_firstname. Waiting for it become active
2022/07/02 20:21:31 Index is 6541687047853166991 now active
2022/07/02 20:21:32 Dropping the secondary index index_age
2022/07/02 20:21:32 Index dropped
2022/07/02 20:21:33 Dropping the secondary index index_firstname
2022/07/02 20:21:34 Index dropped
2022/07/02 20:21:51 Created the secondary index index_age. Waiting for it become active
2022/07/02 20:21:51 Index is 17996046929709083052 now active
2022/07/02 20:22:09 Created the secondary index index_firstname. Waiting for it become active
2022/07/02 20:22:09 Index is 4894322308818442111 now active
2022/07/02 20:22:10 Dropping the secondary index index_age
2022/07/02 20:22:11 Index dropped
2022/07/02 20:22:12 Dropping the secondary index index_firstname
2022/07/02 20:22:12 Index dropped
2022/07/02 20:22:30 Created the secondary index index_age. Waiting for it become active
2022/07/02 20:22:30 Index is 16720411828438534858 now active
2022/07/02 20:22:48 Created the secondary index index_firstname. Waiting for it become active
2022/07/02 20:22:48 Index is 7331908526291125825 now active
2022/07/02 20:22:49 Dropping the secondary index index_age
2022/07/02 20:22:49 Index dropped
2022/07/02 20:22:50 Dropping the secondary index index_firstname
2022/07/02 20:22:50 Index dropped
2022-07-02T20:22:56.165+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:58018->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/07/02 20:23:05 Created the secondary index index_age. Waiting for it become active
2022/07/02 20:23:05 Index is 12248777482314476188 now active
2022/07/02 20:23:18 Created the secondary index index_firstname. Waiting for it become active
2022/07/02 20:23:18 Index is 12902174632795222499 now active
2022/07/02 20:23:20 Dropping the secondary index index_age
2022/07/02 20:23:20 Index dropped
2022/07/02 20:23:21 Dropping the secondary index index_firstname
2022/07/02 20:23:21 Index dropped
--- PASS: TestScanWithConcurrentIndexOps (223.09s)
=== RUN   TestConcurrentScans_SameIndex
2022/07/02 20:23:22 In TestConcurrentScans_SameIndex()
2022/07/02 20:23:22 Generating JSON docs
2022-07-02T20:23:22.225+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:58016->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/07/02 20:23:27 Setting initial JSON docs in KV
2022/07/02 20:24:41 All indexers are active
2022/07/02 20:24:41 Creating a 2i
2022/07/02 20:24:41 Index found:  index_company
2022/07/02 20:24:41 In Range Scan for Thread 6: 
2022/07/02 20:24:41 In Range Scan for Thread 1: 
2022/07/02 20:24:41 In Range Scan for Thread 4: 
2022/07/02 20:24:41 In Range Scan for Thread 2: 
2022/07/02 20:24:41 In Range Scan for Thread 5: 
2022/07/02 20:24:41 In Range Scan for Thread 3: 
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 2: : Index index_company Bucket default
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 4: : Index index_company Bucket default
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 5: : Index index_company Bucket default
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 6: : Index index_company Bucket default
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022/07/02 20:24:41 ListAllSecondaryIndexes() for Thread 3: : Index index_company Bucket default
2022-07-02T20:26:41.836+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60246->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-07-02T20:26:42.013+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60242->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-07-02T20:26:42.086+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60248->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-07-02T20:26:42.236+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60240->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-07-02T20:26:42.309+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60244->127.0.0.1:9106: use of closed network connection. Kill Pipe.
--- PASS: TestConcurrentScans_SameIndex (200.40s)
=== RUN   TestConcurrentScans_MultipleIndexes
2022/07/02 20:26:42 In TestConcurrentScans_MultipleIndexes()
2022-07-02T20:26:42.623+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:60250->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/07/02 20:26:42 Generating JSON docs
2022/07/02 20:26:47 Setting initial JSON docs in KV
2022/07/02 20:27:57 All indexers are active
2022/07/02 20:27:57 Creating multiple indexes
2022/07/02 20:27:57 Index found:  index_company
2022/07/02 20:28:19 Created the secondary index index_age. Waiting for it become active
2022/07/02 20:28:19 Index is 16376686038267748855 now active
2022/07/02 20:28:43 Created the secondary index index_firstname. Waiting for it become active
2022/07/02 20:28:43 Index is 13558266172321137027 now active
2022/07/02 20:28:43 In Range Scan for Thread 3: 
2022/07/02 20:28:43 In Range Scan for Thread 1: 
2022/07/02 20:28:43 In Range Scan
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 3: : Index index_age Bucket default
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 3: : Index index_company Bucket default
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 3: : Index index_firstname Bucket default
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 1: : Index index_firstname Bucket default
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 1: : Index index_age Bucket default
2022/07/02 20:28:43 ListAllSecondaryIndexes() for Thread 1: : Index index_company Bucket default
2022-07-02T20:30:44.309+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34086->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022-07-02T20:30:45.216+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34082->127.0.0.1:9106: use of closed network connection. Kill Pipe.
--- PASS: TestConcurrentScans_MultipleIndexes (242.63s)
=== RUN   TestMutationsWithMultipleIndexBuilds
2022/07/02 20:30:45 In TestMutationsWithMultipleIndexBuilds()
2022/07/02 20:30:45 In DropAllSecondaryIndexes()
2022/07/02 20:30:45 Index found:  index_firstname
2022-07-02T20:30:45.256+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = read tcp 127.0.0.1:34084->127.0.0.1:9106: use of closed network connection. Kill Pipe.
2022/07/02 20:30:45 Dropped index index_firstname
2022/07/02 20:30:45 Index found:  index_company
2022/07/02 20:30:45 Dropped index index_company
2022/07/02 20:30:45 Index found:  index_age
2022/07/02 20:30:45 Dropped index index_age
2022/07/02 20:30:45 Generating JSON docs
2022/07/02 20:30:50 Setting initial JSON docs in KV
2022/07/02 20:32:02 Created the secondary index index_primary. Waiting for it become active
2022/07/02 20:32:02 Index is 12031586642988547526 now active
2022/07/02 20:32:02 Creating multiple indexes in deferred mode
2022/07/02 20:32:03 Build Indexes and wait for indexes to become active: [index_company index_age index_firstname index_lastname]
2022/07/02 20:32:03 Build command issued for the deferred indexes [index_company index_age index_firstname index_lastname]
2022/07/02 20:32:03 Waiting for the index index_company to become active
2022/07/02 20:32:03 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:04 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:05 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:06 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:07 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:08 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:09 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:10 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:11 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:12 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:13 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:14 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:15 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:16 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:17 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:18 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:19 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:20 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:21 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:22 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:23 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:24 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:25 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:26 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:27 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:28 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:29 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:30 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:31 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:32 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:33 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:34 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:35 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:36 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:37 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:38 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:39 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:40 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:41 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:42 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:43 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:44 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:45 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:46 Waiting for index 2006582701198031691 to go active ...
2022/07/02 20:32:47 Index is 2006582701198031691 now active
2022/07/02 20:32:47 Waiting for the index index_age to become active
2022/07/02 20:32:47 Index is 10559923536675919622 now active
2022/07/02 20:32:47 Waiting for the index index_firstname to become active
2022/07/02 20:32:47 Index is 18224433810480307737 now active
2022/07/02 20:32:47 Waiting for the index index_lastname to become active
2022/07/02 20:32:47 Index is 11165828526060517209 now active
--- PASS: TestMutationsWithMultipleIndexBuilds (121.97s)
PASS
ok  	github.com/couchbase/indexing/secondary/tests/largedatatests	1059.945s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 61904    0 61904    0     0  9425k      0 --:--:-- --:--:-- --:--:--  9.8M

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.16.6
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.16.6
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> chronicle (compile)
[100%] Built target chronicle
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target golang_tls_test using Go 1.17.4
[100%] Built target golang_tls_test
[100%] Building Go target ns_goport using Go 1.18.1
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.1
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.1
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.1
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.1
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-07-02 20:33:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-02 20:33:33 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-02 20:33:39 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-02 20:33:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:57 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-07-02 20:33:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '21', 'memoryTotal': 15466930176, 'memoryFree': 5383041024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:33:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:33:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 5203406848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:33:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-02 20:33:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:33:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 5084696576, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:33:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:33:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 4992348160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:33:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-02 20:33:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:34:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:34:00 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-02 20:34:00 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-02 20:34:00 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-07-02 20:34:00 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:34:05 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-02 20:34:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-02 20:34:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.04 seconds to create bucket default
2022-07-02 20:34:06 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-02 20:34:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:34:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:34:07 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:07 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.02977939274388, 'mem_free': 13557641216, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 40.77990674972534
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-02 20:34:11 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-02 20:34:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-02 20:34:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:17 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-07-02 20:34:17 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:34:19 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-07-02 20:34:19 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:19 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-02 20:34:19 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 110.154857ms
2022-07-02 20:34:49 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '131572c3-4469-459d-85f0-fb7772824606', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '110.154857ms', 'executionTime': '110.034057ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-02 20:34:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 409.960425ms
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.054686ms
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.392666ms
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.47945ms
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.61437863901474, 'mem_free': 13533700096, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:34:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-02 20:34:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-02 20:34:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.589s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.61437863901474, 'mem_free': 13533700096, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:34:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-02 20:35:01 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-02 20:35:01 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '88', 'memoryTotal': 15466930176, 'memoryFree': 13518184448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-02 20:35:02 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '83', 'memoryTotal': 15466930176, 'memoryFree': 13520285696, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:02 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '83', 'memoryTotal': 15466930176, 'memoryFree': 13521080320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '82', 'memoryTotal': 15466930176, 'memoryFree': 13521207296, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-02 20:35:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-02 20:35:03 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-02 20:35:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-02 20:35:08 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-02 20:35:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-07-02 20:35:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-02 20:35:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-02 20:35:09 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-02 20:35:09 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-02 20:35:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-02 20:35:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.03 seconds to create bucket default
2022-07-02 20:35:09 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-02 20:35:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:35:10 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:35:11 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.00294486605219, 'mem_free': 13504425984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 19.285188913345337
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-02 20:35:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-02 20:35:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-02 20:35:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:35:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:35:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:35:21 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-07-02 20:35:21 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-02 20:35:23 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-07-02 20:35:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:35:23 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-02 20:35:23 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.973971ms
2022-07-02 20:35:53 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'e5160724-a60e-41d1-a3f0-fc9d70db2cdc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '21.973971ms', 'executionTime': '21.846246ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-02 20:35:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 421.229829ms
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 11.98564ms
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-02 20:35:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:35:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee2602bd10758f47a79cc39e9fd572154fjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-07-02 20:35:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee2602bd10758f47a79cc39e9fd572154fjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-07-02 20:35:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.066916ms
2022-07-02 20:35:55 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee2602bd10758f47a79cc39e9fd572154fjob_title) USING GSI
2022-07-02 20:35:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee2602bd10758f47a79cc39e9fd572154fjob_title) USING GSI
2022-07-02 20:35:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee2602bd10758f47a79cc39e9fd572154fjob_title%29+USING+GSI
2022-07-02 20:35:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.824643ms
2022-07-02 20:35:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee2602bd10758f47a79cc39e9fd572154fjob_title'
2022-07-02 20:35:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2602bd10758f47a79cc39e9fd572154fjob_title%27
2022-07-02 20:35:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.937435ms
2022-07-02 20:35:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee2602bd10758f47a79cc39e9fd572154fjob_title'
2022-07-02 20:35:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2602bd10758f47a79cc39e9fd572154fjob_title%27
2022-07-02 20:35:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.801234ms
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.974195ms
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '8af888d2-d4dc-4021-93a3-0ad2631e3b80', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2602bd10758f47a79cc39e9fd572154fjob_title', 'index_id': '11d38ff68b7d657a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.974195ms', 'executionTime': '2.891561ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:35:59 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-07-02 20:35:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 70.084142ms
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-07-02 20:36:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-07-02 20:36:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:36:01 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee2602bd10758f47a79cc39e9fd572154fjob_title'
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2602bd10758f47a79cc39e9fd572154fjob_title%27
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.050003ms
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee2602bd10758f47a79cc39e9fd572154fjob_title ON default USING GSI
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee2602bd10758f47a79cc39e9fd572154fjob_title+ON+default+USING+GSI
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.825284ms
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee2602bd10758f47a79cc39e9fd572154fjob_title'
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2602bd10758f47a79cc39e9fd572154fjob_title%27
2022-07-02 20:36:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.84316ms
2022-07-02 20:36:02 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '2689d352-2bd4-4eef-9c5a-d5598150e793', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.84316ms', 'executionTime': '1.683734ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.150943ms
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 67.114501ms
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.73651041137964, 'mem_free': 13385228288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-02 20:36:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-02 20:36:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-02 20:36:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-02 20:36:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-02 20:36:09 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-02 20:36:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 73.156s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_2'}
[2022-07-02 20:36:09,226] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:09,227] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:09,487] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:09,493] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:36:09,501] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:36:09,502] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-07-02 20:36:09,502] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:36:10,232] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:36:10,235] - [task:161] INFO -  {'uptime': '156', 'memoryTotal': 15466930176, 'memoryFree': 13426499584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:36:10,237] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:36:10,238] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:36:10,238] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:36:10,243] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:36:10,250] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:36:10,250] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:36:10,254] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:36:10,255] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:36:10,255] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:36:10,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:36:10,258] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:36:10,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:10,262] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:10,534] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:10,535] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:36:10,579] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:36:10,580] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:36:10,584] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:36:10,585] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:36:10,611] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:36:10,615] - [task:161] INFO -  {'uptime': '154', 'memoryTotal': 15466930176, 'memoryFree': 13371781120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:36:10,617] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:36:10,621] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:36:10,621] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:36:10,625] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:36:10,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:10,631] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:10,898] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:10,900] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:36:10,947] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:36:10,948] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:36:10,953] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:36:10,953] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:36:10,973] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:36:10,976] - [task:161] INFO -  {'uptime': '153', 'memoryTotal': 15466930176, 'memoryFree': 13372710912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:36:10,978] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:36:10,982] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:36:10,982] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:36:10,985] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:36:10,990] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:10,990] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:11,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:11,262] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:36:11,306] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:36:11,306] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:36:11,311] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:36:11,312] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:36:11,334] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:36:11,337] - [task:161] INFO -  {'uptime': '152', 'memoryTotal': 15466930176, 'memoryFree': 13394698240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:36:11,339] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:36:11,344] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:36:11,344] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:36:11,346] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:36:11,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:11,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:11,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:11,617] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:36:11,661] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:36:11,662] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:36:11,666] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:36:11,667] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:36:11,683] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:36:11,789] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:36:16,795] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:36:16,807] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:36:16,811] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:16,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:17,081] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:17,083] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:36:17,127] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:36:17,128] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:36:17,128] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:36:17,699] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:36:17,730] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:36:17,731] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:36:18,295] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:36:18,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:36:18,384] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:36:18,386] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:36:18,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:18,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:18,678] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:18,683] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:18,683] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:18,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:18,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:18,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:19,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:19,217] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:19,217] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:19,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:23,290] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:36:23,290] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.04350223736373, 'mem_free': 13422043136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:36:23,290] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:36:23,291] - [basetestcase:467] INFO - Time to execute basesetup : 14.066376209259033
[2022-07-02 20:36:23,294] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:36:23,295] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:36:23,301] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:36:23,302] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:36:23,308] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:36:23,308] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:36:23,314] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:36:23,314] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:36:23,318] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:36:23,335] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:36:23,336] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:36:23,336] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:36:28,344] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:36:28,348] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:36:28,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:36:28,630] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:36:29,563] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:36:29,586] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:36:31,713] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:36:31,730] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-02 20:36:32,333] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:36:32,333] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:36:32,333] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:37:02,343] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:37:02,348] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:37:02,350] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:37:02,371] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.258703ms
[2022-07-02 20:37:02,371] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b265a0d0-f0a0-4ae3-b47d-bf789eed16ad', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.258703ms', 'executionTime': '19.11509ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:37:02,371] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:37:02,374] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:37:02,376] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:37:02,735] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 356.835287ms
[2022-07-02 20:37:02,735] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:37:02,746] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:37:02,751] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:37:02,767] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.519921ms
[2022-07-02 20:37:02,811] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:02,815] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:37:02,830] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:37:02,830] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:37:02,844] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:37:02,858] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:03,811] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6b750682725d44fb82de3fa7bd2c35d6job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:37:03,813] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6b750682725d44fb82de3fa7bd2c35d6job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:37:03,846] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.450628ms
[2022-07-02 20:37:03,847] - [base_gsi:282] INFO - BUILD INDEX on default(employee6b750682725d44fb82de3fa7bd2c35d6job_title) USING GSI
[2022-07-02 20:37:04,853] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee6b750682725d44fb82de3fa7bd2c35d6job_title) USING GSI
[2022-07-02 20:37:04,856] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee6b750682725d44fb82de3fa7bd2c35d6job_title%29+USING+GSI
[2022-07-02 20:37:04,879] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.419945ms
[2022-07-02 20:37:05,886] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b750682725d44fb82de3fa7bd2c35d6job_title'
[2022-07-02 20:37:05,889] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b750682725d44fb82de3fa7bd2c35d6job_title%27
[2022-07-02 20:37:05,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.642026ms
[2022-07-02 20:37:06,903] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b750682725d44fb82de3fa7bd2c35d6job_title'
[2022-07-02 20:37:06,905] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b750682725d44fb82de3fa7bd2c35d6job_title%27
[2022-07-02 20:37:06,913] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.101153ms
[2022-07-02 20:37:07,381] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-02 20:37:07,407] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:37:08,403] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:37:08,916] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:08,925] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:37:08,928] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-02 20:37:08,933] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.184736ms
[2022-07-02 20:37:08,933] - [task:3216] INFO - {'requestID': '80d00cf9-331c-4b39-9046-6e297d76c6c1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6b750682725d44fb82de3fa7bd2c35d6job_title', 'index_id': '1b1abfe6442c771d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.184736ms', 'executionTime': '3.113472ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-02 20:37:08,933] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:08,934] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:08,934] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:37:08,934] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:37:08,935] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-02 20:37:08,935] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:37:08,935] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:37:08,935] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:37:08,936] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:37:09,935] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:09,941] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:37:09,944] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-02 20:37:10,011] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.548177ms
[2022-07-02 20:37:10,011] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:37:10,012] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:37:10,795] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:10,796] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:37:11,802] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b750682725d44fb82de3fa7bd2c35d6job_title'
[2022-07-02 20:37:11,805] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b750682725d44fb82de3fa7bd2c35d6job_title%27
[2022-07-02 20:37:11,815] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.000714ms
[2022-07-02 20:37:11,817] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee6b750682725d44fb82de3fa7bd2c35d6job_title ON default USING GSI
[2022-07-02 20:37:11,820] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee6b750682725d44fb82de3fa7bd2c35d6job_title+ON+default+USING+GSI
[2022-07-02 20:37:11,863] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.93018ms
[2022-07-02 20:37:11,870] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b750682725d44fb82de3fa7bd2c35d6job_title'
[2022-07-02 20:37:11,873] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b750682725d44fb82de3fa7bd2c35d6job_title%27
[2022-07-02 20:37:11,876] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.636085ms
[2022-07-02 20:37:11,877] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ed9c1d41-d12d-4d19-86b9-abcaa688d8d6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.636085ms', 'executionTime': '1.575555ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:37:11,885] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:11,889] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:11,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:12,242] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:12,250] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:37:12,250] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:37:12,308] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:37:12,316] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:12,319] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:12,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:12,655] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:12,663] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:37:12,663] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:37:12,726] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:37:12,734] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:12,735] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:37:12,735] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:37:12,737] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:37:12,741] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:37:12,750] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.003274ms
[2022-07-02 20:37:12,753] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:37:12,755] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:37:12,810] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.289146ms
[2022-07-02 20:37:12,827] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:37:12,827] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.74960292287164, 'mem_free': 13297139712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:37:12,827] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:37:12,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:12,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:13,199] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:13,204] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:13,204] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:13,673] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:13,680] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:13,681] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:14,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:14,312] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:14,312] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:14,934] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:19,049] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:37:19,073] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:37:20,093] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:37:20,096] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:37:20,097] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:37:20,102] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:37:20,110] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:37:20,116] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:37:20,117] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:37:20,127] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:37:20,128] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:37:20,130] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,142] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:37:20,143] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,146] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:37:20,149] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:37:20,150] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,155] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:37:20,162] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:37:20,163] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,167] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:37:20,170] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:37:20,171] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,177] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:37:20,178] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:37:20,178] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:37:20,179] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 70.960s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_3'}
[2022-07-02 20:37:20,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:20,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:20,570] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:20,576] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:37:20,583] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:37:20,583] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-07-02 20:37:20,584] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:37:21,244] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:37:21,248] - [task:161] INFO -  {'uptime': '227', 'memoryTotal': 15466930176, 'memoryFree': 13354602496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:37:21,250] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:37:21,250] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:37:21,251] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:37:21,257] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:37:21,263] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:37:21,264] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:37:21,267] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:37:21,268] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:37:21,269] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:37:21,269] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:37:21,272] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:37:21,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:21,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:21,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:21,608] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:37:21,658] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:37:21,659] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:37:21,664] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:37:21,665] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:37:21,693] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:37:21,697] - [task:161] INFO -  {'uptime': '224', 'memoryTotal': 15466930176, 'memoryFree': 13301600256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:37:21,699] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:37:21,705] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:37:21,705] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:37:21,708] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:37:21,712] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:21,712] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:22,058] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:22,059] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:37:22,119] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:37:22,121] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:37:22,126] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:37:22,127] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:37:22,152] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:37:22,155] - [task:161] INFO -  {'uptime': '224', 'memoryTotal': 15466930176, 'memoryFree': 13295460352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:37:22,158] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:37:22,163] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:37:22,164] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:37:22,166] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:37:22,172] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:22,172] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:22,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:22,495] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:37:22,546] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:37:22,548] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:37:22,553] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:37:22,554] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:37:22,574] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:37:22,577] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13353963520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:37:22,579] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:37:22,583] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:37:22,584] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:37:22,586] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:37:22,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:22,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:22,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:22,912] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:37:22,969] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:37:22,970] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:37:22,975] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:37:22,976] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:37:22,993] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:37:23,112] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:37:28,114] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:37:28,128] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:37:28,133] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:28,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:28,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:28,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:37:28,514] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:37:28,515] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:37:28,515] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:37:29,010] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:37:29,041] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:37:29,042] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:37:29,604] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:37:29,729] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:37:30,329] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:37:30,383] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:37:30,393] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:30,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:30,751] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:30,756] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:30,756] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:31,075] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:31,080] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:31,080] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:31,471] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:31,478] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:31,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:32,042] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:36,105] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:37:36,105] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.79196488813585, 'mem_free': 13358727168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:37:36,105] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:37:36,106] - [basetestcase:467] INFO - Time to execute basesetup : 15.868074893951416
[2022-07-02 20:37:36,110] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:37:36,110] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:37:36,117] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:37:36,117] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:37:36,124] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:37:36,124] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:37:36,130] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:37:36,130] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:37:36,134] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:36,153] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:37:36,154] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:37:36,154] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:37:41,167] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:37:41,170] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:37:41,170] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:37:41,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:37:42,417] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:37:42,439] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:37:44,631] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:37:44,653] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-02 20:37:45,445] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:37:45,446] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:37:45,446] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:38:15,476] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:38:15,482] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:38:15,484] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:38:15,505] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.448589ms
[2022-07-02 20:38:15,506] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a1c05f2f-591c-4459-a346-8ea3a719aa5c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.448589ms', 'executionTime': '19.382503ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:38:15,506] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:38:15,509] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:38:15,511] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:38:15,894] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 380.648548ms
[2022-07-02 20:38:15,894] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:38:15,900] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:38:15,910] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:38:15,923] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.377397ms
[2022-07-02 20:38:15,957] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:15,960] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:38:15,975] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:38:15,978] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:38:15,991] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:38:16,007] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:16,957] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2d9e2c67e2624b01996c0a34579a1688job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:38:16,960] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2d9e2c67e2624b01996c0a34579a1688job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:38:17,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.990408ms
[2022-07-02 20:38:17,008] - [base_gsi:282] INFO - BUILD INDEX on default(employee2d9e2c67e2624b01996c0a34579a1688job_title) USING GSI
[2022-07-02 20:38:18,015] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee2d9e2c67e2624b01996c0a34579a1688job_title) USING GSI
[2022-07-02 20:38:18,017] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee2d9e2c67e2624b01996c0a34579a1688job_title%29+USING+GSI
[2022-07-02 20:38:18,039] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.21806ms
[2022-07-02 20:38:19,045] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d9e2c67e2624b01996c0a34579a1688job_title'
[2022-07-02 20:38:19,047] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d9e2c67e2624b01996c0a34579a1688job_title%27
[2022-07-02 20:38:19,055] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.999016ms
[2022-07-02 20:38:20,061] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d9e2c67e2624b01996c0a34579a1688job_title'
[2022-07-02 20:38:20,064] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d9e2c67e2624b01996c0a34579a1688job_title%27
[2022-07-02 20:38:20,073] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.292757ms
[2022-07-02 20:38:20,557] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-02 20:38:20,580] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:38:21,315] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:38:22,075] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:22,083] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:38:22,085] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-02 20:38:22,089] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.385158ms
[2022-07-02 20:38:22,089] - [task:3216] INFO - {'requestID': '254f3eed-077f-4d63-8ad4-315ebbd4cf95', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2d9e2c67e2624b01996c0a34579a1688job_title', 'index_id': '37c5a5110f0cea00', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.385158ms', 'executionTime': '2.313209ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-02 20:38:22,090] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:22,090] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:22,091] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:38:22,091] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:38:22,091] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-02 20:38:22,091] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:38:22,092] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:38:22,092] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:38:22,092] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:38:23,092] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:23,096] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:38:23,099] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-02 20:38:23,165] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.293127ms
[2022-07-02 20:38:23,165] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:38:23,166] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:38:23,966] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:23,966] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:38:24,973] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d9e2c67e2624b01996c0a34579a1688job_title'
[2022-07-02 20:38:24,975] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d9e2c67e2624b01996c0a34579a1688job_title%27
[2022-07-02 20:38:24,986] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.169212ms
[2022-07-02 20:38:24,988] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2d9e2c67e2624b01996c0a34579a1688job_title ON default USING GSI
[2022-07-02 20:38:24,990] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2d9e2c67e2624b01996c0a34579a1688job_title+ON+default+USING+GSI
[2022-07-02 20:38:25,033] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.48697ms
[2022-07-02 20:38:25,042] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d9e2c67e2624b01996c0a34579a1688job_title'
[2022-07-02 20:38:25,045] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d9e2c67e2624b01996c0a34579a1688job_title%27
[2022-07-02 20:38:25,048] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.60655ms
[2022-07-02 20:38:25,048] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '257ec9fc-b5ba-4057-82d5-fc8aa5487a90', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.60655ms', 'executionTime': '1.537561ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:38:25,057] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:25,061] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:25,061] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:25,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:25,485] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:38:25,485] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:38:25,555] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:38:25,562] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:25,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:25,565] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:25,968] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:25,976] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:38:25,976] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:38:26,045] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:38:26,056] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:26,057] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:38:26,057] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:38:26,059] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:38:26,061] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:38:26,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.671674ms
[2022-07-02 20:38:26,073] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:38:26,075] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:38:26,141] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.677106ms
[2022-07-02 20:38:26,151] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:38:26,151] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.93654246672991, 'mem_free': 13237510144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:38:26,151] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:38:26,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:26,156] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:26,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:26,568] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:26,569] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:26,958] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:26,963] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:26,963] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:27,406] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:27,412] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:27,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:28,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:33,120] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:38:33,141] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:38:33,526] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:38:33,530] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:38:33,530] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:38:33,537] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:38:33,543] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:38:33,549] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:38:33,550] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:38:33,559] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:38:33,560] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:38:33,562] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:33,576] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:38:33,576] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:33,580] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:38:33,583] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:38:33,583] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:33,586] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:38:33,589] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:38:33,589] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:33,592] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:38:33,594] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:38:33,594] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:33,597] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:38:33,597] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:38:33,598] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:38:33,598] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 73.366s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_4'}
[2022-07-02 20:38:33,657] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:33,657] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:34,051] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:34,057] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:38:34,067] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:38:34,067] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-07-02 20:38:34,068] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:38:34,662] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:38:34,666] - [task:161] INFO -  {'uptime': '301', 'memoryTotal': 15466930176, 'memoryFree': 13242413056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:38:34,668] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:38:34,668] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:38:34,669] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:38:34,677] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:38:34,686] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:38:34,686] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:38:34,689] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:38:34,690] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:38:34,690] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:38:34,690] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:38:34,692] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:38:34,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:34,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:35,129] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:35,130] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:38:35,200] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:38:35,201] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:38:35,207] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:38:35,207] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:38:35,239] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:38:35,242] - [task:161] INFO -  {'uptime': '299', 'memoryTotal': 15466930176, 'memoryFree': 13313089536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:38:35,244] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:38:35,251] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:38:35,251] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:38:35,253] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:38:35,256] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:35,256] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:35,654] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:35,655] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:38:35,721] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:38:35,722] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:38:35,727] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:38:35,728] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:38:35,751] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:38:35,754] - [task:161] INFO -  {'uptime': '299', 'memoryTotal': 15466930176, 'memoryFree': 13313236992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:38:35,756] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:38:35,761] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:38:35,761] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:38:35,763] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:38:35,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:35,770] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:36,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:36,174] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:38:36,236] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:38:36,237] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:38:36,242] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:38:36,244] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:38:36,265] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:38:36,269] - [task:161] INFO -  {'uptime': '293', 'memoryTotal': 15466930176, 'memoryFree': 13220884480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:38:36,271] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:38:36,276] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:38:36,276] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:38:36,278] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:38:36,282] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:36,282] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:36,687] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:36,689] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:38:36,752] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:38:36,753] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:38:36,758] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:38:36,759] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:38:36,776] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:38:36,907] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:38:41,908] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:38:41,920] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:38:41,924] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:41,924] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:42,323] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:42,325] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:38:42,383] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:38:42,384] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:38:42,385] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:38:42,793] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:38:42,825] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:38:42,825] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:38:43,484] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:38:43,526] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:38:43,570] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:38:43,574] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:38:43,582] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:43,582] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:43,972] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:43,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:43,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:44,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:44,346] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:44,346] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:44,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:44,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:44,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:45,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:50,019] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:38:50,020] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.40502382086613, 'mem_free': 13313204224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:38:50,020] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:38:50,020] - [basetestcase:467] INFO - Time to execute basesetup : 16.365828275680542
[2022-07-02 20:38:50,025] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:38:50,025] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:38:50,032] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:38:50,032] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:38:50,038] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:38:50,039] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:38:50,044] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:38:50,045] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:38:50,049] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:50,064] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:38:50,065] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:38:50,065] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:38:55,073] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:38:55,077] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:38:55,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:38:55,464] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:38:56,484] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:38:56,510] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:38:58,461] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:38:58,477] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-02 20:38:59,196] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:38:59,196] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:38:59,197] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:39:29,223] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:39:29,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:39:29,231] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:39:29,254] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.85357ms
[2022-07-02 20:39:29,254] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '14a20dfb-b76e-4e42-8cf5-236f21241a93', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.85357ms', 'executionTime': '20.775142ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:39:29,254] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:39:29,256] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:39:29,259] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:39:29,619] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 357.129884ms
[2022-07-02 20:39:29,620] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:39:29,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:39:29,634] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:39:29,644] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.148005ms
[2022-07-02 20:39:29,673] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:39:29,677] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:39:29,692] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:39:29,693] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:39:29,707] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:39:29,721] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:39:30,672] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeed18b1c6905704553898eb06307332867job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:39:30,675] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeed18b1c6905704553898eb06307332867job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:39:30,702] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.564049ms
[2022-07-02 20:39:30,703] - [base_gsi:282] INFO - BUILD INDEX on default(employeed18b1c6905704553898eb06307332867job_title) USING GSI
[2022-07-02 20:39:31,710] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeed18b1c6905704553898eb06307332867job_title) USING GSI
[2022-07-02 20:39:31,712] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeed18b1c6905704553898eb06307332867job_title%29+USING+GSI
[2022-07-02 20:39:31,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.282533ms
[2022-07-02 20:39:32,745] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed18b1c6905704553898eb06307332867job_title'
[2022-07-02 20:39:32,748] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed18b1c6905704553898eb06307332867job_title%27
[2022-07-02 20:39:32,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.727279ms
[2022-07-02 20:39:33,765] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed18b1c6905704553898eb06307332867job_title'
[2022-07-02 20:39:33,767] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed18b1c6905704553898eb06307332867job_title%27
[2022-07-02 20:39:33,774] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.326892ms
[2022-07-02 20:39:34,291] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-02 20:39:34,315] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:39:35,165] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:39:35,207] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-07-02 20:39:35,207] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:39:35,233] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-07-02 20:39:35,234] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-07-02 20:39:35,234] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2026005696, 0, b'')
[2022-07-02 20:39:35,776] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:35,780] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:39:35,783] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-02 20:39:35,787] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.44167ms
[2022-07-02 20:39:35,787] - [task:3216] INFO - {'requestID': '04e9317d-3788-400f-bdcc-82c94ab7ec17', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeed18b1c6905704553898eb06307332867job_title', 'index_id': 'de495ee6d07bc945', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.44167ms', 'executionTime': '2.360809ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-02 20:39:35,787] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:35,788] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:35,788] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:39:35,789] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:39:35,789] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-02 20:39:35,789] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:39:35,789] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:39:35,790] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:39:35,790] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:39:36,789] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:36,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:39:36,796] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-02 20:39:36,867] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.739222ms
[2022-07-02 20:39:36,868] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:39:36,868] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:39:37,650] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:37,650] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:39:38,656] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed18b1c6905704553898eb06307332867job_title'
[2022-07-02 20:39:38,658] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed18b1c6905704553898eb06307332867job_title%27
[2022-07-02 20:39:38,667] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.436856ms
[2022-07-02 20:39:38,669] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeed18b1c6905704553898eb06307332867job_title ON default USING GSI
[2022-07-02 20:39:38,671] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeed18b1c6905704553898eb06307332867job_title+ON+default+USING+GSI
[2022-07-02 20:39:38,726] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.391221ms
[2022-07-02 20:39:38,735] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeed18b1c6905704553898eb06307332867job_title'
[2022-07-02 20:39:38,737] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeed18b1c6905704553898eb06307332867job_title%27
[2022-07-02 20:39:38,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.602224ms
[2022-07-02 20:39:38,740] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fcbbcf67-6847-4062-bd5b-59f11f734b00', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.602224ms', 'executionTime': '1.546933ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:39:38,749] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:39:38,753] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:38,753] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:39,243] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:39,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:39:39,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:39:39,327] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:39:39,335] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:39:39,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:39,338] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:39,787] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:39,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:39:39,798] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:39:39,874] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:39:39,882] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:39:39,882] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:39:39,882] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:39:39,884] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:39:39,887] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:39:39,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.036001ms
[2022-07-02 20:39:39,897] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:39:39,900] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:39:39,952] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.209459ms
[2022-07-02 20:39:39,964] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:39:39,965] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.68212221515684, 'mem_free': 13208145920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:39:39,965] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:39:39,973] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:39,974] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:40,423] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:40,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:40,429] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:40,883] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:40,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:40,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:41,462] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:41,469] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:41,469] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:42,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:47,663] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:39:47,691] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:39:48,280] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:39:48,283] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:39:48,283] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:39:48,290] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:39:48,297] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:39:48,304] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:39:48,305] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:39:48,314] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:39:48,315] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:39:48,317] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,330] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:39:48,330] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,335] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:39:48,337] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:39:48,337] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,341] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:39:48,343] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:39:48,343] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,346] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:39:48,348] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:39:48,348] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,352] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:39:48,352] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:39:48,352] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:39:48,353] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 74.703s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_5'}
[2022-07-02 20:39:48,409] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:48,409] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:48,852] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:48,859] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:39:48,868] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:39:48,868] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-07-02 20:39:48,869] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:39:49,414] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:39:49,418] - [task:161] INFO -  {'uptime': '376', 'memoryTotal': 15466930176, 'memoryFree': 13290774528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:39:49,421] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:39:49,421] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:39:49,421] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:39:49,432] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:39:49,439] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:39:49,439] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:39:49,442] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:39:49,443] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:39:49,444] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:39:49,444] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:39:49,446] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:39:49,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:49,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:49,879] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:49,880] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:39:49,945] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:39:49,946] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:39:49,952] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:39:49,952] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:39:49,980] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:39:49,983] - [task:161] INFO -  {'uptime': '374', 'memoryTotal': 15466930176, 'memoryFree': 13292187648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:39:49,985] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:39:49,990] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:39:49,990] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:39:49,993] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:39:49,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:49,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:50,427] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:50,428] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:39:50,496] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:39:50,497] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:39:50,502] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:39:50,503] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:39:50,524] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:39:50,527] - [task:161] INFO -  {'uptime': '369', 'memoryTotal': 15466930176, 'memoryFree': 13291667456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:39:50,529] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:39:50,534] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:39:50,534] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:39:50,536] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:39:50,543] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:50,543] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:50,981] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:50,983] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:39:51,050] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:39:51,051] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:39:51,056] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:39:51,057] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:39:51,079] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:39:51,082] - [task:161] INFO -  {'uptime': '368', 'memoryTotal': 15466930176, 'memoryFree': 13292023808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:39:51,085] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:39:51,090] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:39:51,090] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:39:51,093] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:39:51,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:51,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:51,516] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:51,517] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:39:51,585] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:39:51,586] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:39:51,590] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:39:51,591] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:39:51,610] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:39:51,729] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:39:56,730] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:39:56,742] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:39:56,747] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:56,747] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:57,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:57,224] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:39:57,296] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:39:57,297] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:39:57,297] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:39:57,628] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:39:57,658] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:39:57,659] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:39:58,230] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:39:58,281] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:39:58,452] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:39:58,455] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:39:58,462] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:58,462] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:58,895] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:58,900] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:58,901] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:59,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:59,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:59,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:39:59,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:39:59,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:39:59,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:40:00,751] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:40:05,937] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:40:05,937] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.34252723547266, 'mem_free': 13291655168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:40:05,938] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:40:05,938] - [basetestcase:467] INFO - Time to execute basesetup : 17.53100895881653
[2022-07-02 20:40:05,942] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:40:05,942] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:40:05,949] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:40:05,950] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:40:05,956] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:40:05,956] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:40:05,962] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:40:05,962] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:40:05,966] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:05,977] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:40:05,977] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:40:05,977] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:40:10,990] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:40:10,994] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:40:10,994] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:40:11,423] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:40:12,385] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:40:12,407] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:40:14,532] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:40:14,550] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-02 20:40:15,597] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:15,597] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:40:15,597] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:40:45,627] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:40:45,632] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:40:45,634] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:40:45,655] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.364287ms
[2022-07-02 20:40:45,655] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'efbbd5ea-b055-4fa8-8882-c21ee4898c92', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.364287ms', 'executionTime': '19.29352ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:40:45,656] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:40:45,658] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:40:45,660] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:40:46,044] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 381.567806ms
[2022-07-02 20:40:46,044] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:40:46,059] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:40:46,068] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:40:46,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.53132ms
[2022-07-02 20:40:46,140] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:46,150] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:40:46,168] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:40:46,168] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:40:46,181] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:40:46,198] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:47,134] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee392c4f5041c141849cf05fc4efe2ac91job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:40:47,137] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee392c4f5041c141849cf05fc4efe2ac91job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:40:47,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.480423ms
[2022-07-02 20:40:47,174] - [base_gsi:282] INFO - BUILD INDEX on default(employee392c4f5041c141849cf05fc4efe2ac91job_title) USING GSI
[2022-07-02 20:40:48,179] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee392c4f5041c141849cf05fc4efe2ac91job_title) USING GSI
[2022-07-02 20:40:48,182] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee392c4f5041c141849cf05fc4efe2ac91job_title%29+USING+GSI
[2022-07-02 20:40:48,204] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.277295ms
[2022-07-02 20:40:49,211] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee392c4f5041c141849cf05fc4efe2ac91job_title'
[2022-07-02 20:40:49,213] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee392c4f5041c141849cf05fc4efe2ac91job_title%27
[2022-07-02 20:40:49,223] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.363627ms
[2022-07-02 20:40:50,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee392c4f5041c141849cf05fc4efe2ac91job_title'
[2022-07-02 20:40:50,231] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee392c4f5041c141849cf05fc4efe2ac91job_title%27
[2022-07-02 20:40:50,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.797388ms
[2022-07-02 20:40:50,627] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:40:50,791] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:40:53,354] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:40:54,245] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:54,252] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:40:54,254] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-02 20:40:54,259] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.23509ms
[2022-07-02 20:40:54,259] - [task:3216] INFO - {'requestID': '552db34d-1644-449f-8661-8c14a521635f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee392c4f5041c141849cf05fc4efe2ac91job_title', 'index_id': '2efefa8ec27b9b06', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.23509ms', 'executionTime': '3.157346ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-02 20:40:54,259] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:54,260] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:54,260] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:40:54,261] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:40:54,261] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-02 20:40:54,261] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:40:54,261] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:40:54,262] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:40:54,262] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:40:55,261] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:55,266] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:40:55,268] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-02 20:40:55,376] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 92.621686ms
[2022-07-02 20:40:55,376] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:40:55,377] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:40:56,930] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:56,930] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:40:57,938] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee392c4f5041c141849cf05fc4efe2ac91job_title'
[2022-07-02 20:40:57,940] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee392c4f5041c141849cf05fc4efe2ac91job_title%27
[2022-07-02 20:40:57,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.474839ms
[2022-07-02 20:40:57,952] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee392c4f5041c141849cf05fc4efe2ac91job_title ON default USING GSI
[2022-07-02 20:40:57,954] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee392c4f5041c141849cf05fc4efe2ac91job_title+ON+default+USING+GSI
[2022-07-02 20:40:58,003] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.416144ms
[2022-07-02 20:40:58,012] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee392c4f5041c141849cf05fc4efe2ac91job_title'
[2022-07-02 20:40:58,014] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee392c4f5041c141849cf05fc4efe2ac91job_title%27
[2022-07-02 20:40:58,017] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.588722ms
[2022-07-02 20:40:58,017] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2aa4e325-585c-40c4-837a-de7c3cd609ab', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.588722ms', 'executionTime': '1.476481ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:40:58,027] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:58,030] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:40:58,030] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:40:58,592] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:40:58,604] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:40:58,604] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:40:58,699] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:40:58,706] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:58,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:40:58,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:40:59,244] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:40:59,252] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:40:59,253] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:40:59,339] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:40:59,346] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:40:59,347] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:40:59,347] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:40:59,349] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:40:59,351] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:40:59,359] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.72676ms
[2022-07-02 20:40:59,361] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:40:59,363] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:40:59,418] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.108996ms
[2022-07-02 20:40:59,428] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:40:59,428] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.82802125464827, 'mem_free': 13129486336, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:40:59,428] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:40:59,434] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:40:59,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:40:59,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:40:59,970] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:40:59,970] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:00,661] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:00,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:00,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:01,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:01,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:01,616] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:02,555] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:08,256] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:41:08,282] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:41:09,033] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:41:09,037] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:41:09,037] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:41:09,043] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:41:09,049] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:41:09,056] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:41:09,057] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:41:09,066] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:41:09,067] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:41:09,069] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,082] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:41:09,082] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,086] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:41:09,088] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:41:09,089] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,092] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:41:09,094] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:41:09,094] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,097] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:41:09,099] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:41:09,099] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,103] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:41:09,103] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:41:09,103] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:41:09,104] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 80.702s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_6'}
[2022-07-02 20:41:09,168] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:09,168] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:09,705] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:09,712] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:41:09,720] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:41:09,721] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-07-02 20:41:09,721] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:41:10,172] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:41:10,176] - [task:161] INFO -  {'uptime': '456', 'memoryTotal': 15466930176, 'memoryFree': 13232762880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:41:10,178] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:41:10,179] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:41:10,179] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:41:10,185] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:41:10,203] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:41:10,204] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:41:10,207] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:41:10,208] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:41:10,208] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:41:10,209] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:41:10,211] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:41:10,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:10,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:10,744] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:10,746] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:41:10,836] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:41:10,837] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:41:10,842] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:41:10,842] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:41:10,870] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:41:10,873] - [task:161] INFO -  {'uptime': '455', 'memoryTotal': 15466930176, 'memoryFree': 13234601984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:41:10,876] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:41:10,881] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:41:10,882] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:41:10,884] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:41:10,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:10,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:11,454] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:11,455] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:41:11,538] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:41:11,539] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:41:11,545] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:41:11,545] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:41:11,566] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:41:11,569] - [task:161] INFO -  {'uptime': '455', 'memoryTotal': 15466930176, 'memoryFree': 13234216960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:41:11,572] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:41:11,576] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:41:11,577] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:41:11,579] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:41:11,586] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:11,586] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:12,112] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:12,113] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:41:12,208] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:41:12,209] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:41:12,216] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:41:12,217] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:41:12,241] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:41:12,245] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13233930240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:41:12,247] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:41:12,252] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:41:12,253] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:41:12,255] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:41:12,259] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:12,260] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:12,788] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:12,789] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:41:12,872] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:41:12,873] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:41:12,878] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:41:12,879] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:41:12,897] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:41:13,006] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:41:18,011] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:41:18,023] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:41:18,029] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:18,029] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:18,536] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:18,537] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:41:18,618] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:41:18,619] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:41:18,619] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:41:18,914] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:41:18,943] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:41:18,944] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:41:19,510] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:41:19,550] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:41:20,228] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:41:20,231] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:41:20,238] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:20,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:20,788] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:20,793] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:20,793] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:21,427] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:21,436] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:21,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:22,333] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:22,341] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:22,341] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:23,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:28,859] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:41:28,859] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.51309665346356, 'mem_free': 13237440512, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:41:28,860] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:41:28,860] - [basetestcase:467] INFO - Time to execute basesetup : 19.694413423538208
[2022-07-02 20:41:28,864] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:41:28,865] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:41:28,871] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:41:28,871] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:41:28,877] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:41:28,877] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:41:28,884] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:41:28,884] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:41:28,888] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:41:28,910] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:41:28,911] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:41:28,911] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:41:33,924] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:41:33,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:41:33,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:41:34,431] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:41:35,548] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:41:35,572] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:41:38,306] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:41:38,325] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-02 20:41:39,846] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:41:39,846] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:41:39,847] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:42:09,847] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:42:09,853] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:42:09,856] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:42:09,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.367491ms
[2022-07-02 20:42:09,877] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8b98a598-a183-4807-a6c9-90c2757227a0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.367491ms', 'executionTime': '19.287845ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:42:09,877] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:42:09,880] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:42:09,882] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:42:10,238] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 354.663036ms
[2022-07-02 20:42:10,239] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:42:10,250] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:42:10,255] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:42:10,264] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.630325ms
[2022-07-02 20:42:10,299] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:42:10,305] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:42:10,320] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:42:10,321] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:42:10,342] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:42:10,354] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:42:11,296] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefcca8da980754f50b871b9c2fd848be6job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:42:11,298] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefcca8da980754f50b871b9c2fd848be6job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:42:11,332] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.509643ms
[2022-07-02 20:42:11,332] - [base_gsi:282] INFO - BUILD INDEX on default(employeefcca8da980754f50b871b9c2fd848be6job_title) USING GSI
[2022-07-02 20:42:12,338] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeefcca8da980754f50b871b9c2fd848be6job_title) USING GSI
[2022-07-02 20:42:12,341] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeefcca8da980754f50b871b9c2fd848be6job_title%29+USING+GSI
[2022-07-02 20:42:12,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.842297ms
[2022-07-02 20:42:13,372] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefcca8da980754f50b871b9c2fd848be6job_title'
[2022-07-02 20:42:13,374] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefcca8da980754f50b871b9c2fd848be6job_title%27
[2022-07-02 20:42:13,383] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.536718ms
[2022-07-02 20:42:14,389] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefcca8da980754f50b871b9c2fd848be6job_title'
[2022-07-02 20:42:14,392] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefcca8da980754f50b871b9c2fd848be6job_title%27
[2022-07-02 20:42:14,398] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.238971ms
[2022-07-02 20:42:14,779] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-02 20:42:14,813] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:42:16,074] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:42:16,487] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-02 20:42:16,519] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:42:17,125] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:42:17,793] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:42:17,822] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:42:21,294] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:42:21,417] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:21,425] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:42:21,427] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-02 20:42:21,432] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.927197ms
[2022-07-02 20:42:21,432] - [task:3216] INFO - {'requestID': '1ab65c7a-ff11-4c78-8725-73ce64d3562e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefcca8da980754f50b871b9c2fd848be6job_title', 'index_id': '45a755e1b6aa7531', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.927197ms', 'executionTime': '2.861284ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-02 20:42:21,432] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:21,433] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:21,433] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:42:21,433] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:42:21,434] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-02 20:42:21,434] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:42:21,434] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:42:21,435] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:42:21,435] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-02 20:42:22,434] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:22,440] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-02 20:42:22,442] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-02 20:42:22,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 112.500085ms
[2022-07-02 20:42:22,570] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:42:22,571] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:42:24,151] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:24,152] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-02 20:42:25,159] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefcca8da980754f50b871b9c2fd848be6job_title'
[2022-07-02 20:42:25,161] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefcca8da980754f50b871b9c2fd848be6job_title%27
[2022-07-02 20:42:25,172] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.320882ms
[2022-07-02 20:42:25,176] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeefcca8da980754f50b871b9c2fd848be6job_title ON default USING GSI
[2022-07-02 20:42:25,179] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeefcca8da980754f50b871b9c2fd848be6job_title+ON+default+USING+GSI
[2022-07-02 20:42:25,219] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.468233ms
[2022-07-02 20:42:25,231] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefcca8da980754f50b871b9c2fd848be6job_title'
[2022-07-02 20:42:25,234] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefcca8da980754f50b871b9c2fd848be6job_title%27
[2022-07-02 20:42:25,237] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.530933ms
[2022-07-02 20:42:25,237] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b4664c19-c927-4c68-a7b0-fd6af3ed9995', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.530933ms', 'executionTime': '1.473992ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:42:25,245] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:42:25,249] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:25,249] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:25,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:25,949] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:42:25,949] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:42:26,056] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:42:26,064] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:42:26,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:26,067] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:26,746] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:26,756] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:42:26,757] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:42:26,863] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:42:26,871] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:42:26,871] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:42:26,872] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:42:26,874] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:42:26,876] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:42:26,885] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.423466ms
[2022-07-02 20:42:26,888] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:42:26,890] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:42:26,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.987914ms
[2022-07-02 20:42:26,970] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:42:26,970] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.7539755991586, 'mem_free': 13088722944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:42:26,970] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:42:26,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:26,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:27,661] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:27,667] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:27,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:28,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:28,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:28,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:29,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:29,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:29,558] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:30,694] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:37,936] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:42:37,963] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:42:38,383] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:42:38,386] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:42:38,386] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:42:38,393] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:42:38,399] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:42:38,406] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:42:38,408] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:42:38,416] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:42:38,416] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:42:38,418] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:38,430] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:42:38,431] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:38,434] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:42:38,437] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:42:38,437] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:38,441] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:42:38,443] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:42:38,443] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:38,446] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:42:38,449] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:42:38,449] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:38,452] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:42:38,452] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-02 20:42:38,453] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:42:38,453] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 89.293s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_7'}
[2022-07-02 20:42:38,549] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:38,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:39,187] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:39,193] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:42:39,202] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:42:39,203] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-07-02 20:42:39,204] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:42:39,553] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:42:39,557] - [task:161] INFO -  {'uptime': '546', 'memoryTotal': 15466930176, 'memoryFree': 13173002240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:42:39,559] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:42:39,559] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:42:39,560] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:42:39,568] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:42:39,579] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:42:39,579] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:42:39,582] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:42:39,583] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:42:39,584] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:42:39,584] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:42:39,586] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:42:39,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:39,591] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:40,235] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:40,236] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:42:40,331] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:42:40,332] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:42:40,337] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:42:40,338] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:42:40,365] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:42:40,369] - [task:161] INFO -  {'uptime': '545', 'memoryTotal': 15466930176, 'memoryFree': 13172895744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:42:40,372] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:42:40,377] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:42:40,377] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:42:40,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:42:40,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:40,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:41,006] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:41,007] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:42:41,104] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:42:41,105] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:42:41,110] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:42:41,111] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:42:41,137] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:42:41,140] - [task:161] INFO -  {'uptime': '540', 'memoryTotal': 15466930176, 'memoryFree': 13173403648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:42:41,143] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:42:41,147] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:42:41,147] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:42:41,152] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:42:41,158] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:41,158] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:41,807] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:41,808] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:42:41,904] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:42:41,905] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:42:41,910] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:42:41,911] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:42:41,932] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:42:41,935] - [task:161] INFO -  {'uptime': '539', 'memoryTotal': 15466930176, 'memoryFree': 13173510144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:42:41,938] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:42:41,942] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:42:41,942] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:42:41,946] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:42:41,950] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:41,950] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:42,575] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:42,576] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:42:42,672] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:42:42,673] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:42:42,678] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:42:42,679] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:42:42,695] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:42:42,807] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:42:47,812] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:42:47,824] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:42:47,829] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:47,829] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:48,487] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:48,489] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:42:48,586] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:42:48,586] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:42:48,587] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:42:48,712] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:42:48,744] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:42:48,744] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:42:49,311] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:42:49,353] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:42:50,009] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:42:50,013] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-02 20:42:50,021] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:50,021] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:50,723] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:50,728] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:50,728] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:51,620] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:51,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:51,630] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:52,735] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:42:52,745] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:42:52,745] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:42:53,851] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:00,419] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:43:00,419] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.94901677683925, 'mem_free': 13189345280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:43:00,419] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:43:00,420] - [basetestcase:467] INFO - Time to execute basesetup : 21.873291492462158
[2022-07-02 20:43:00,424] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:43:00,425] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:43:00,431] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:43:00,431] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:43:00,437] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:43:00,438] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:43:00,443] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:43:00,443] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:43:00,448] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:00,461] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:43:00,462] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:43:00,462] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:43:05,474] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:43:05,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:05,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:06,075] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:07,233] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:43:07,259] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:43:09,870] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:43:09,902] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:09,902] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:43:09,903] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:43:39,932] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:43:39,938] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:43:39,940] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:43:39,962] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.718846ms
[2022-07-02 20:43:39,962] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1d38a0c9-d610-4214-80d1-f505b958f038', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.718846ms', 'executionTime': '19.634361ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:43:39,962] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:43:39,965] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:43:39,967] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:43:40,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 339.370395ms
[2022-07-02 20:43:40,309] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:43:40,317] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:43:40,319] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:43:40,329] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.05356ms
[2022-07-02 20:43:40,364] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:40,368] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:43:40,394] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:43:40,395] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:43:40,410] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:43:40,411] - [base_gsi:326] INFO - []
[2022-07-02 20:43:41,362] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4d558dbe4d744d639174266e64640f5ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:43:41,364] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4d558dbe4d744d639174266e64640f5ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:43:41,395] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.07402ms
[2022-07-02 20:43:41,402] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4d558dbe4d744d639174266e64640f5ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:43:41,405] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4d558dbe4d744d639174266e64640f5ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:43:41,437] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.640982ms
[2022-07-02 20:43:41,438] - [base_gsi:282] INFO - BUILD INDEX on default(employee4d558dbe4d744d639174266e64640f5ajob_title,employee4d558dbe4d744d639174266e64640f5ajoin_yr) USING GSI
[2022-07-02 20:43:42,444] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee4d558dbe4d744d639174266e64640f5ajob_title,employee4d558dbe4d744d639174266e64640f5ajoin_yr) USING GSI
[2022-07-02 20:43:42,447] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee4d558dbe4d744d639174266e64640f5ajob_title%2Cemployee4d558dbe4d744d639174266e64640f5ajoin_yr%29+USING+GSI
[2022-07-02 20:43:42,485] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.880442ms
[2022-07-02 20:43:43,493] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajob_title'
[2022-07-02 20:43:43,495] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajob_title%27
[2022-07-02 20:43:43,505] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.436475ms
[2022-07-02 20:43:44,512] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajob_title'
[2022-07-02 20:43:44,514] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajob_title%27
[2022-07-02 20:43:44,521] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.607549ms
[2022-07-02 20:43:44,523] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajoin_yr'
[2022-07-02 20:43:44,526] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajoin_yr%27
[2022-07-02 20:43:44,528] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.359335ms
[2022-07-02 20:43:45,535] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajob_title'
[2022-07-02 20:43:45,537] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajob_title%27
[2022-07-02 20:43:45,547] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.861823ms
[2022-07-02 20:43:45,549] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4d558dbe4d744d639174266e64640f5ajob_title ON default USING GSI
[2022-07-02 20:43:45,551] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4d558dbe4d744d639174266e64640f5ajob_title+ON+default+USING+GSI
[2022-07-02 20:43:45,585] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.553728ms
[2022-07-02 20:43:45,591] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajoin_yr'
[2022-07-02 20:43:45,594] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajoin_yr%27
[2022-07-02 20:43:45,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.503604ms
[2022-07-02 20:43:45,600] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4d558dbe4d744d639174266e64640f5ajoin_yr ON default USING GSI
[2022-07-02 20:43:45,603] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4d558dbe4d744d639174266e64640f5ajoin_yr+ON+default+USING+GSI
[2022-07-02 20:43:45,655] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.461269ms
[2022-07-02 20:43:45,663] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajob_title'
[2022-07-02 20:43:45,665] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajob_title%27
[2022-07-02 20:43:45,673] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.602637ms
[2022-07-02 20:43:45,673] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fb7ffb57-12c1-48fe-b36b-d58876e946e1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.602637ms', 'executionTime': '5.534467ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:43:45,675] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4d558dbe4d744d639174266e64640f5ajoin_yr'
[2022-07-02 20:43:45,678] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4d558dbe4d744d639174266e64640f5ajoin_yr%27
[2022-07-02 20:43:45,680] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 987.265µs
[2022-07-02 20:43:45,680] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '25bb2502-7abe-484e-b8fa-71df1c660870', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '987.265µs', 'executionTime': '936.914µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:43:45,688] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:45,691] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:45,692] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:46,322] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:46,331] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:43:46,331] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:43:46,433] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:43:46,441] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:46,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:46,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:47,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:47,092] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:43:47,092] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:43:47,198] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:43:47,206] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:43:47,206] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:43:47,206] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:43:47,209] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:43:47,211] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:43:47,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.261354ms
[2022-07-02 20:43:47,223] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:43:47,225] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:43:47,266] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.133402ms
[2022-07-02 20:43:47,295] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:43:47,296] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.741284156521744, 'mem_free': 13080608768, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:43:47,296] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:43:47,301] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:47,301] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:47,934] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:47,939] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:47,939] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:48,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:48,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:48,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:49,604] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:49,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:49,613] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:50,691] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:57,495] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-07-02 20:43:57,521] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:43:58,211] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:43:58,214] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:43:58,215] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:43:58,221] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:43:58,228] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:43:58,234] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:43:58,235] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:43:58,243] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:43:58,245] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:43:58,247] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,259] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:43:58,259] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,262] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:43:58,265] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:43:58,265] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,268] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:43:58,270] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:43:58,270] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,273] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:43:58,276] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:43:58,276] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,279] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:43:58,279] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-02 20:43:58,279] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:43:58,280] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 79.739s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_8'}
[2022-07-02 20:43:58,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:58,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:58,948] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:58,957] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:43:58,965] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:43:58,965] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-07-02 20:43:58,966] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:43:59,326] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:43:59,330] - [task:161] INFO -  {'uptime': '625', 'memoryTotal': 15466930176, 'memoryFree': 13203709952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:43:59,332] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:43:59,332] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:43:59,333] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:43:59,341] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:43:59,350] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:43:59,351] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:43:59,354] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:43:59,355] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:43:59,355] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:43:59,355] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:43:59,358] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:43:59,363] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:43:59,363] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:43:59,964] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:43:59,965] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:44:00,066] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:44:00,067] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:44:00,072] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:44:00,072] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:44:00,100] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:44:00,104] - [task:161] INFO -  {'uptime': '620', 'memoryTotal': 15466930176, 'memoryFree': 13202702336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:44:00,107] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:44:00,112] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:44:00,113] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:44:00,116] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:44:00,120] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:00,120] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:00,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:00,754] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:44:00,848] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:44:00,849] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:44:00,854] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:44:00,855] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:44:00,877] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:44:00,880] - [task:161] INFO -  {'uptime': '620', 'memoryTotal': 15466930176, 'memoryFree': 13202374656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:44:00,883] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:44:00,888] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:44:00,888] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:44:00,891] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:44:00,897] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:00,897] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:01,546] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:01,548] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:44:01,646] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:44:01,648] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:44:01,653] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:44:01,654] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:44:01,673] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:44:01,676] - [task:161] INFO -  {'uptime': '619', 'memoryTotal': 15466930176, 'memoryFree': 13202595840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:44:01,679] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:44:01,684] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:44:01,684] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:44:01,686] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:44:01,690] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:01,691] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:02,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:02,320] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:44:02,412] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:44:02,413] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:44:02,418] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:44:02,419] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:44:02,437] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:44:02,569] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:44:07,571] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:44:07,582] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:44:07,586] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:07,586] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:08,217] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:08,218] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:44:08,325] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:44:08,326] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:44:08,326] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:44:08,455] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:44:08,484] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:44:08,485] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:44:09,053] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:44:09,100] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:44:09,145] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:44:09,149] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-02 20:44:09,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:09,156] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:09,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:09,803] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:09,803] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:10,681] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:10,693] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:10,693] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:11,778] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:11,791] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:11,791] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:12,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:19,291] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:44:19,292] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.12059248984088, 'mem_free': 13200474112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:44:19,292] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:44:19,292] - [basetestcase:467] INFO - Time to execute basesetup : 20.973257064819336
[2022-07-02 20:44:19,298] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:44:19,298] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:44:19,305] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:44:19,305] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:44:19,312] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:44:19,312] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:44:19,319] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:44:19,319] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:44:19,323] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:44:19,337] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:44:19,338] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:44:19,338] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:44:24,349] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:44:24,353] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:44:24,353] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:44:24,951] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:44:26,122] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:44:26,148] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:44:28,749] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:44:28,778] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:44:28,779] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:44:28,779] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:44:58,804] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:44:58,809] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:44:58,812] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:44:58,833] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.180387ms
[2022-07-02 20:44:58,833] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ef2084d5-395c-4652-ab4c-479e497f634d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.180387ms', 'executionTime': '19.111031ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:44:58,833] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:44:58,836] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:44:58,838] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:44:59,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 401.6178ms
[2022-07-02 20:44:59,242] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:44:59,254] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:44:59,259] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:44:59,267] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.591654ms
[2022-07-02 20:44:59,291] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:44:59,300] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:44:59,318] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:44:59,319] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:44:59,334] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:44:59,338] - [base_gsi:326] INFO - []
[2022-07-02 20:45:00,292] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:45:00,294] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:45:00,325] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.938482ms
[2022-07-02 20:45:00,326] - [base_gsi:282] INFO - BUILD INDEX on default(employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr) USING GSI
[2022-07-02 20:45:01,332] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr) USING GSI
[2022-07-02 20:45:01,335] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%29+USING+GSI
[2022-07-02 20:45:01,357] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.236001ms
[2022-07-02 20:45:02,365] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr'
[2022-07-02 20:45:02,368] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%27
[2022-07-02 20:45:02,376] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.178322ms
[2022-07-02 20:45:03,383] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr'
[2022-07-02 20:45:03,385] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%27
[2022-07-02 20:45:03,394] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.917875ms
[2022-07-02 20:45:04,401] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr'
[2022-07-02 20:45:04,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%27
[2022-07-02 20:45:04,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.526788ms
[2022-07-02 20:45:04,413] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr ON default USING GSI
[2022-07-02 20:45:04,416] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr+ON+default+USING+GSI
[2022-07-02 20:45:04,466] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.298781ms
[2022-07-02 20:45:04,473] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr'
[2022-07-02 20:45:04,476] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ab892da1ad8424cb437e74734b2a332job_title_join_yr%27
[2022-07-02 20:45:04,479] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.591276ms
[2022-07-02 20:45:04,479] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5640d8c8-fbff-4ca0-903b-e1d51e304785', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.591276ms', 'executionTime': '1.51875ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:45:04,487] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:45:04,490] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:04,491] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:05,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:05,222] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:45:05,223] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:45:05,329] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:45:05,337] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:45:05,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:05,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:06,017] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:06,025] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:45:06,025] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:45:06,140] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:45:06,149] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:45:06,149] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:45:06,149] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:45:06,151] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:45:06,154] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:45:06,164] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.870568ms
[2022-07-02 20:45:06,166] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:45:06,168] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:45:06,208] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.345999ms
[2022-07-02 20:45:06,230] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:45:06,231] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.263686046259303, 'mem_free': 13074972672, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:45:06,231] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:45:06,236] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:06,236] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:06,870] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:06,875] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:06,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:07,561] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:07,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:07,649] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:08,772] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:08,784] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:08,784] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:09,882] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:16,697] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-07-02 20:45:16,722] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:45:17,853] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:45:17,856] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:45:17,856] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:45:17,862] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:45:17,868] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:45:17,874] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:45:17,875] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:45:17,884] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:45:17,885] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:45:17,887] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:17,899] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:45:17,899] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:17,903] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:45:17,905] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:45:17,906] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:17,911] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:45:17,915] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:45:17,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:17,920] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:45:17,923] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:45:17,923] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:17,926] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:45:17,926] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-02 20:45:17,926] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:45:17,927] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 79.613s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_9'}
[2022-07-02 20:45:17,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:17,970] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:18,602] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:18,608] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:45:18,617] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:45:18,617] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-07-02 20:45:18,618] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:45:18,973] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:45:18,977] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13188132864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:45:18,979] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:45:18,979] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:45:18,980] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:45:18,990] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:45:18,995] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:45:18,996] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:45:18,998] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:45:18,999] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:45:19,000] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:45:19,000] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:45:19,003] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:45:19,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:19,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:19,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:19,639] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:45:19,733] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:45:19,735] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:45:19,739] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:45:19,740] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:45:19,764] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:45:19,768] - [task:161] INFO -  {'uptime': '701', 'memoryTotal': 15466930176, 'memoryFree': 13186998272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:45:19,770] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:45:19,777] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:45:19,777] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:45:19,779] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:45:19,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:19,786] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:20,409] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:20,410] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:45:20,504] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:45:20,506] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:45:20,510] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:45:20,512] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:45:20,531] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:45:20,534] - [task:161] INFO -  {'uptime': '701', 'memoryTotal': 15466930176, 'memoryFree': 13186850816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:45:20,536] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:45:20,540] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:45:20,540] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:45:20,543] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:45:20,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:20,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:21,206] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:21,207] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:45:21,300] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:45:21,301] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:45:21,306] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:45:21,307] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:45:21,331] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:45:21,334] - [task:161] INFO -  {'uptime': '700', 'memoryTotal': 15466930176, 'memoryFree': 13186924544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:45:21,337] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:45:21,341] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:45:21,342] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:45:21,344] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:45:21,348] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:21,348] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:21,967] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:21,968] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:45:22,063] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:45:22,064] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:45:22,069] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:45:22,070] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:45:22,088] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:45:22,219] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:45:27,225] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:45:27,235] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:45:27,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:27,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:27,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:27,920] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:45:28,022] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:45:28,023] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:45:28,024] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:45:28,106] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:45:28,136] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:45:28,136] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:45:28,709] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:45:28,753] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:45:28,796] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:45:28,800] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-02 20:45:28,808] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:28,808] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:29,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:29,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:29,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:30,403] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:30,414] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:30,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:31,512] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:31,521] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:31,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:32,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:39,192] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:45:39,193] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.59316213183125, 'mem_free': 13185388544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:45:39,193] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:45:39,193] - [basetestcase:467] INFO - Time to execute basesetup : 21.226375579833984
[2022-07-02 20:45:39,197] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:45:39,197] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:45:39,205] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:45:39,205] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:45:39,211] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:45:39,211] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:45:39,218] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:45:39,218] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:45:39,223] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:45:39,235] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:45:39,235] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:45:39,236] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:45:44,249] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:45:44,252] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:45:44,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:45:44,871] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:45:45,912] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:45:46,088] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:45:48,666] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:45:48,696] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:45:48,696] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:45:48,699] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:46:18,727] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:46:18,733] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:46:18,736] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:46:18,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.138047ms
[2022-07-02 20:46:18,758] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '089f32bd-701c-445c-928e-c52f317db872', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.138047ms', 'executionTime': '20.071655ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:46:18,759] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:46:18,761] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:46:18,763] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:46:19,130] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 365.477508ms
[2022-07-02 20:46:19,131] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:46:19,159] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:46:19,163] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:46:19,174] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.747042ms
[2022-07-02 20:46:19,222] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:19,227] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:46:19,246] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:46:19,246] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:46:19,260] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:46:19,272] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:20,212] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6435471719a04ab7b1d482be28e9eb8bjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:46:20,215] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:46:20,262] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.726495ms
[2022-07-02 20:46:20,263] - [base_gsi:282] INFO - BUILD INDEX on default(`employee6435471719a04ab7b1d482be28e9eb8bjoin_yr`) USING GSI
[2022-07-02 20:46:21,269] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee6435471719a04ab7b1d482be28e9eb8bjoin_yr`) USING GSI
[2022-07-02 20:46:21,272] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%60%29+USING+GSI
[2022-07-02 20:46:21,299] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.150479ms
[2022-07-02 20:46:21,310] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6435471719a04ab7b1d482be28e9eb8bjoin_yr'
[2022-07-02 20:46:21,317] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%27
[2022-07-02 20:46:21,329] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.530537ms
[2022-07-02 20:46:22,335] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6435471719a04ab7b1d482be28e9eb8bjoin_yr'
[2022-07-02 20:46:22,338] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%27
[2022-07-02 20:46:22,348] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.598669ms
[2022-07-02 20:46:23,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6435471719a04ab7b1d482be28e9eb8bjoin_yr'
[2022-07-02 20:46:23,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%27
[2022-07-02 20:46:23,366] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.564309ms
[2022-07-02 20:46:24,372] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:46:24,374] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-02 20:46:24,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.134841ms
[2022-07-02 20:46:24,380] - [base_gsi:504] INFO - {'requestID': 'e7c9a396-140c-44d5-bb71-386fcf5fcfa8', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6435471719a04ab7b1d482be28e9eb8bjoin_yr', 'index_id': '597bab1fe41ace05', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '4.134841ms', 'executionTime': '4.066834ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-02 20:46:24,380] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:46:24,381] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:46:24,381] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:46:24,381] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:46:24,382] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:46:24,383] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:46:24,383] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:46:24,384] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:46:24,444] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:46:24,459] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:46:24,461] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-02 20:46:24,561] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 86.622265ms
[2022-07-02 20:46:24,561] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:46:24,562] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:46:26,515] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:26,517] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6435471719a04ab7b1d482be28e9eb8bjoin_yr'
[2022-07-02 20:46:26,519] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6435471719a04ab7b1d482be28e9eb8bjoin_yr%27
[2022-07-02 20:46:26,525] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.765809ms
[2022-07-02 20:46:26,525] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fe80e52b-9b27-4e22-bfb4-1ce73a7bd5f1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.765809ms', 'executionTime': '4.653586ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:46:26,533] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:26,536] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:26,536] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:27,219] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:27,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:46:27,227] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:46:27,381] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:46:27,388] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:27,391] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:27,391] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:28,051] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:28,061] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:46:28,062] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:46:28,173] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:46:28,181] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:46:28,181] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:46:28,182] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:46:28,184] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:46:28,186] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:46:28,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.809018ms
[2022-07-02 20:46:28,194] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '95db80b5-c4a1-42a0-b40a-ec6c6b37ad15', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.809018ms', 'executionTime': '5.742197ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:46:28,197] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:46:28,198] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.759279956573863, 'mem_free': 13057724416, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:46:28,198] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:46:28,202] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:28,202] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:28,869] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:28,873] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:28,873] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:29,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:29,640] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:29,640] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:30,773] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:30,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:30,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:31,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:38,976] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-07-02 20:46:38,990] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:46:38,996] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:46:39,002] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:46:39,008] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:46:39,009] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:46:39,015] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:46:39,016] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:46:39,018] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,032] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:46:39,032] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,036] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:46:39,038] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:46:39,038] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,041] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:46:39,043] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:46:39,043] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,046] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:46:39,048] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:46:39,049] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,052] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:46:39,052] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-02 20:46:39,052] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:46:39,053] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 81.091s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_10'}
[2022-07-02 20:46:39,091] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:39,091] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:39,760] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:39,766] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:46:39,775] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:46:39,775] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-07-02 20:46:39,776] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:46:40,096] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:46:40,099] - [task:161] INFO -  {'uptime': '784', 'memoryTotal': 15466930176, 'memoryFree': 13033291776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:46:40,101] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:46:40,101] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:46:40,102] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:46:40,112] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:46:40,123] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:46:40,124] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:46:40,126] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:46:40,127] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:46:40,127] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:46:40,128] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:46:40,130] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:46:40,133] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:40,134] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:40,818] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:40,819] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:46:40,923] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:46:40,923] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:46:40,928] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:46:40,929] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:46:40,962] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:46:40,966] - [task:161] INFO -  {'uptime': '781', 'memoryTotal': 15466930176, 'memoryFree': 13181083648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:46:40,969] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:46:40,976] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:46:40,976] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:46:40,978] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:46:40,984] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:40,985] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:41,657] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:41,658] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:46:41,757] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:46:41,759] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:46:41,764] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:46:41,765] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:46:41,790] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:46:41,794] - [task:161] INFO -  {'uptime': '781', 'memoryTotal': 15466930176, 'memoryFree': 13181243392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:46:41,796] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:46:41,800] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:46:41,801] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:46:41,803] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:46:41,807] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:41,808] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:42,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:42,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:46:42,570] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:46:42,571] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:46:42,576] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:46:42,577] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:46:42,598] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:46:42,602] - [task:161] INFO -  {'uptime': '780', 'memoryTotal': 15466930176, 'memoryFree': 13180129280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:46:42,604] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:46:42,612] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:46:42,612] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:46:42,614] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:46:42,618] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:42,619] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:43,312] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:43,313] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:46:43,411] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:46:43,412] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:46:43,416] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:46:43,417] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:46:43,434] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:46:43,540] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:46:48,542] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:46:48,553] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:46:48,557] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:48,557] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:49,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:49,256] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:46:49,361] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:46:49,362] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:46:49,362] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:46:49,451] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:46:49,483] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:46:49,484] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:46:50,054] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:46:50,100] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:46:50,143] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:46:50,146] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-07-02 20:46:50,153] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:50,154] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:50,847] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:50,852] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:50,852] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:51,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:51,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:51,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:53,040] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:46:53,051] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:46:53,051] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:46:54,211] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:01,013] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:47:01,014] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.37519458770206, 'mem_free': 13178257408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:47:01,014] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:47:01,014] - [basetestcase:467] INFO - Time to execute basesetup : 21.925574779510498
[2022-07-02 20:47:01,018] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:47:01,019] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:47:01,025] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:47:01,025] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:47:01,032] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:47:01,032] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:47:01,039] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:47:01,039] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:47:01,044] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:01,057] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:47:01,057] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:47:01,058] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:47:06,069] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:47:06,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:06,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:06,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:07,751] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:47:07,937] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:47:10,489] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:47:10,517] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:10,517] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:47:10,517] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:47:40,537] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:47:40,542] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:47:40,545] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:47:40,567] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.810053ms
[2022-07-02 20:47:40,567] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '358e5dcd-a678-40db-b5cc-41d77a839aa8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.810053ms', 'executionTime': '20.734281ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:47:40,568] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:47:40,570] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:47:40,572] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:47:40,954] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 379.858102ms
[2022-07-02 20:47:40,954] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:47:40,965] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:47:40,970] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:47:40,979] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.870776ms
[2022-07-02 20:47:41,003] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:41,010] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:47:41,032] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:47:41,032] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:47:41,044] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:47:41,056] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:42,002] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:47:42,004] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:47:42,037] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.105393ms
[2022-07-02 20:47:42,038] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr`) USING GSI
[2022-07-02 20:47:43,044] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr`) USING GSI
[2022-07-02 20:47:43,047] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%60%29+USING+GSI
[2022-07-02 20:47:43,074] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.430887ms
[2022-07-02 20:47:43,085] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr'
[2022-07-02 20:47:43,093] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%27
[2022-07-02 20:47:43,104] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.080422ms
[2022-07-02 20:47:44,111] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr'
[2022-07-02 20:47:44,113] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%27
[2022-07-02 20:47:44,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.749501ms
[2022-07-02 20:47:45,127] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr'
[2022-07-02 20:47:45,129] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%27
[2022-07-02 20:47:45,139] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.355209ms
[2022-07-02 20:47:46,146] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:47:46,148] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-02 20:47:46,153] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.15136ms
[2022-07-02 20:47:46,153] - [base_gsi:504] INFO - {'requestID': '188ce8d2-4477-4218-aa54-dab17d487d52', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr', 'index_id': '91b463ec16fc97e5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.15136ms', 'executionTime': '3.076001ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-02 20:47:46,154] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:47:46,154] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:46,155] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:47:46,155] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:47:46,155] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:46,155] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:46,155] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:46,156] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:46,217] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:46,233] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:47:46,235] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-02 20:47:46,349] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 101.942694ms
[2022-07-02 20:47:46,350] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:47:46,351] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:47:47,529] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-07-02 20:47:47,561] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-07-02 20:47:47,566] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:47:47,569] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-02 20:47:47,572] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.793488ms
[2022-07-02 20:47:47,572] - [base_gsi:504] INFO - {'requestID': 'cb81665e-b94b-4f7f-b1aa-f39b7d25a7ff', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr', 'index_id': '91b463ec16fc97e5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.793488ms', 'executionTime': '1.719391ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-02 20:47:47,572] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:47:47,573] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:47,573] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:47:47,573] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:47:47,573] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:47,573] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:47,574] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:47,574] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:47:47,630] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:47:47,645] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:47:47,647] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-02 20:47:47,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 103.316487ms
[2022-07-02 20:47:47,759] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:47:47,760] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:47:48,967] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr ON default USING GSI
[2022-07-02 20:47:48,969] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr+ON+default+USING+GSI
[2022-07-02 20:47:49,010] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.75104ms
[2022-07-02 20:47:49,017] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr'
[2022-07-02 20:47:49,020] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4f796ac556d943c59cfcdaf8c41cc3bbjoin_yr%27
[2022-07-02 20:47:49,029] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.349752ms
[2022-07-02 20:47:49,029] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '315b3073-1cd0-4f4a-bcfb-7f32d393cf52', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.349752ms', 'executionTime': '5.283451ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:47:49,038] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:49,041] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:49,041] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:49,765] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:49,775] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:47:49,776] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:47:49,885] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:47:49,892] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:49,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:49,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:50,618] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:50,626] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:47:50,626] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:47:50,744] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:47:50,752] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:47:50,752] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:47:50,752] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:47:50,754] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:47:50,757] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:47:50,767] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.535097ms
[2022-07-02 20:47:50,769] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:47:50,773] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:47:50,819] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.929836ms
[2022-07-02 20:47:50,834] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:47:50,834] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.82709886225578, 'mem_free': 13022642176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:47:50,834] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:47:50,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:50,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:51,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:51,564] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:51,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:52,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:52,288] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:52,288] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:53,560] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:47:53,569] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:47:53,569] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:47:54,798] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:02,528] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-07-02 20:48:02,552] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:48:02,916] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:48:02,919] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:48:02,919] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:48:02,925] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:48:02,932] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:48:02,938] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:48:02,939] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:48:02,948] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:48:02,949] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:48:02,951] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:02,965] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:48:02,965] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:02,971] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:48:02,974] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:48:02,974] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:02,978] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:48:02,980] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:48:02,980] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:02,984] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:48:02,986] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:48:02,986] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:02,990] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:48:02,990] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-07-02 20:48:02,991] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:48:02,991] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 83.909s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_11'}
[2022-07-02 20:48:03,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:03,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:03,726] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:03,732] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:48:03,741] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:48:03,741] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-07-02 20:48:03,742] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:48:04,034] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:48:04,038] - [task:161] INFO -  {'uptime': '870', 'memoryTotal': 15466930176, 'memoryFree': 13168881664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:48:04,040] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:48:04,040] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:48:04,041] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:48:04,047] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:48:04,053] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:48:04,054] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:48:04,058] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:48:04,059] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:48:04,059] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:48:04,060] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:48:04,062] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:48:04,066] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:04,069] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:04,779] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:04,780] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:48:04,889] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:48:04,890] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:48:04,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:48:04,896] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:48:04,939] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:48:04,942] - [task:161] INFO -  {'uptime': '867', 'memoryTotal': 15466930176, 'memoryFree': 13167509504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:48:04,944] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:48:04,949] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:48:04,949] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:48:04,953] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:48:04,957] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:04,957] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:05,703] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:05,704] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:48:05,808] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:48:05,809] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:48:05,814] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:48:05,815] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:48:05,841] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:48:05,844] - [task:161] INFO -  {'uptime': '866', 'memoryTotal': 15466930176, 'memoryFree': 13169000448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:48:05,846] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:48:05,850] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:48:05,850] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:48:05,852] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:48:05,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:05,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:06,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:06,549] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:48:06,653] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:48:06,654] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:48:06,660] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:48:06,661] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:48:06,680] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:48:06,682] - [task:161] INFO -  {'uptime': '866', 'memoryTotal': 15466930176, 'memoryFree': 13168521216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:48:06,685] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:48:06,689] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:48:06,690] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:48:06,692] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:48:06,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:06,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:07,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:07,377] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:48:07,478] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:48:07,479] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:48:07,483] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:48:07,484] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:48:07,501] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:48:07,609] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:48:12,612] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:48:12,624] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:48:12,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:12,628] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:13,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:13,361] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:48:13,466] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:48:13,467] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:48:13,467] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:48:13,517] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:48:13,547] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:48:13,547] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:48:14,118] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:48:14,163] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:48:14,206] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:48:14,209] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-02 20:48:14,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:14,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:14,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:14,932] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:14,932] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:15,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:15,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:15,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:17,163] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:17,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:17,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:18,361] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:25,673] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:48:25,674] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.07829451868574, 'mem_free': 13167669248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:48:25,674] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:48:25,674] - [basetestcase:467] INFO - Time to execute basesetup : 22.64608359336853
[2022-07-02 20:48:25,678] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:48:25,679] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:48:25,685] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:48:25,685] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:48:25,692] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:48:25,692] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:48:25,698] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:48:25,698] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:48:25,703] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:48:25,713] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:48:25,714] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:48:25,714] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:48:30,721] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:48:30,725] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:48:30,725] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:48:31,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:48:32,474] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:48:32,499] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:48:35,361] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:48:35,382] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:48:35,382] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:48:35,382] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:49:05,412] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:49:05,418] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:49:05,420] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:49:05,441] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.380981ms
[2022-07-02 20:49:05,441] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '87cac9b5-5304-4689-91bf-e16311eb859e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.380981ms', 'executionTime': '19.313051ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:49:05,441] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:49:05,444] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:49:05,446] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:49:05,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 379.042125ms
[2022-07-02 20:49:05,827] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:49:05,837] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:49:05,856] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:49:05,870] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.119491ms
[2022-07-02 20:49:05,900] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:05,904] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:49:05,926] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:49:05,927] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:49:05,939] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:49:05,954] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:06,898] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee9cdf38df161646c3b774d687030b155djoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-02 20:49:06,901] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee9cdf38df161646c3b774d687030b155djoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-02 20:49:06,933] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.503597ms
[2022-07-02 20:49:06,933] - [base_gsi:282] INFO - BUILD INDEX on default(`employee9cdf38df161646c3b774d687030b155djoin_yr`) USING GSI
[2022-07-02 20:49:07,939] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee9cdf38df161646c3b774d687030b155djoin_yr`) USING GSI
[2022-07-02 20:49:07,942] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee9cdf38df161646c3b774d687030b155djoin_yr%60%29+USING+GSI
[2022-07-02 20:49:07,961] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.300185ms
[2022-07-02 20:49:07,972] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9cdf38df161646c3b774d687030b155djoin_yr'
[2022-07-02 20:49:07,976] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9cdf38df161646c3b774d687030b155djoin_yr%27
[2022-07-02 20:49:07,987] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.430871ms
[2022-07-02 20:49:08,994] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9cdf38df161646c3b774d687030b155djoin_yr'
[2022-07-02 20:49:08,996] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9cdf38df161646c3b774d687030b155djoin_yr%27
[2022-07-02 20:49:09,003] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.116004ms
[2022-07-02 20:49:10,008] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9cdf38df161646c3b774d687030b155djoin_yr'
[2022-07-02 20:49:10,011] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9cdf38df161646c3b774d687030b155djoin_yr%27
[2022-07-02 20:49:10,019] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.352709ms
[2022-07-02 20:49:11,026] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:49:11,029] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-02 20:49:11,033] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.799442ms
[2022-07-02 20:49:11,033] - [base_gsi:504] INFO - {'requestID': '5eb5a65d-c288-4115-8042-efb3beb690a2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee9cdf38df161646c3b774d687030b155djoin_yr', 'index_id': '796075d79deba875', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.799442ms', 'executionTime': '2.730958ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-02 20:49:11,034] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-02 20:49:11,034] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:49:11,034] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-02 20:49:11,034] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-02 20:49:11,035] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:49:11,035] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:49:11,035] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:49:11,035] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-02 20:49:11,090] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-02 20:49:11,105] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-02 20:49:11,107] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-02 20:49:11,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 99.133275ms
[2022-07-02 20:49:11,221] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-02 20:49:11,221] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-02 20:49:12,973] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-02 20:49:15,782] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:49:15,813] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:49:15,813] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:49:16,373] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:49:16,412] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:49:17,094] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:49:17,097] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-02 20:49:19,105] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:49:19,105] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:19,109] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:49:19,114] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:49:19,114] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:19,117] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:49:19,122] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:49:19,122] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:19,125] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:49:19,129] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:49:19,130] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:19,133] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:49:19,137] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:19,139] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9cdf38df161646c3b774d687030b155djoin_yr'
[2022-07-02 20:49:19,141] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9cdf38df161646c3b774d687030b155djoin_yr%27
[2022-07-02 20:49:19,162] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.856736ms
[2022-07-02 20:49:19,163] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fe45eaf5-c2a5-49f2-b0dc-b73abc5a45a5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.856736ms', 'executionTime': '19.792373ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:49:19,165] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee9cdf38df161646c3b774d687030b155djoin_yr'
[2022-07-02 20:49:19,167] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee9cdf38df161646c3b774d687030b155djoin_yr%27
[2022-07-02 20:49:19,169] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 999.277µs
[2022-07-02 20:49:19,170] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e9437e6d-ee6b-42fb-8791-6a1284a9c014', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '999.277µs', 'executionTime': '948.43µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:49:19,173] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:19,184] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:19,186] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:19,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:19,939] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:19,947] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:49:19,947] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:49:20,064] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:49:20,071] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:20,074] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:20,074] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:20,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:20,832] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:49:20,832] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:49:20,956] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:49:20,963] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:20,964] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:49:20,964] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:49:20,966] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:49:20,968] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:49:20,977] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.354738ms
[2022-07-02 20:49:20,977] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2da3652f-f264-49aa-a6ae-c6f983bdb7d0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.354738ms', 'executionTime': '7.288091ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:49:20,977] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:49:20,979] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:49:20,982] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:49:20,984] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 997.456µs
[2022-07-02 20:49:20,984] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3f6b8a1d-30ac-4dc8-b0ae-cca35bde9f46', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '997.456µs', 'executionTime': '956.994µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:49:20,988] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:49:20,988] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.30461680119125, 'mem_free': 13145694208, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:49:20,989] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:49:20,992] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:20,992] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:21,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:21,729] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:21,729] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:22,688] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:22,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:22,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:23,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:23,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:23,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:25,240] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:33,010] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-07-02 20:49:33,034] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:49:33,322] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:49:33,325] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:49:33,325] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:49:33,332] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:49:33,339] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:49:33,346] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:49:33,347] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:49:33,365] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:49:33,366] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:49:33,369] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:33,384] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:49:33,385] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:33,388] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:49:33,390] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:49:33,390] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:33,393] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:49:33,396] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:49:33,396] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:33,399] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:49:33,401] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:49:33,401] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:33,404] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:49:33,404] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-02 20:49:33,404] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:49:33,405] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 90.381s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-02_20-33-29/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-07-02 20:49:33,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:33,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:34,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:34,161] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:49:34,170] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-02 20:49:34,170] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-07-02 20:49:34,171] - [basetestcase:224] INFO - initializing cluster
[2022-07-02 20:49:34,420] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-02 20:49:34,424] - [task:161] INFO -  {'uptime': '961', 'memoryTotal': 15466930176, 'memoryFree': 13145989120, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:49:34,426] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-02 20:49:34,426] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-02 20:49:34,427] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-02 20:49:34,433] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-02 20:49:34,439] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-02 20:49:34,441] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-02 20:49:34,448] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-02 20:49:34,449] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-02 20:49:34,449] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-02 20:49:34,449] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-02 20:49:34,453] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:49:34,465] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:34,465] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:35,203] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:35,204] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:49:35,311] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:49:35,313] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:49:35,319] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:49:35,320] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:49:35,344] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-02 20:49:35,348] - [task:161] INFO -  {'uptime': '957', 'memoryTotal': 15466930176, 'memoryFree': 13145763840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:49:35,351] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:49:35,356] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-02 20:49:35,356] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-02 20:49:35,358] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:49:35,362] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:35,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:36,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:36,081] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:49:36,187] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:49:36,188] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:49:36,193] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:49:36,193] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:49:36,214] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-02 20:49:36,217] - [task:161] INFO -  {'uptime': '957', 'memoryTotal': 15466930176, 'memoryFree': 13145980928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:49:36,220] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:49:36,225] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-02 20:49:36,225] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-02 20:49:36,228] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:49:36,232] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:36,232] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:36,952] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:36,954] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:49:37,060] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:49:37,061] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:49:37,066] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:49:37,066] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:49:37,088] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-02 20:49:37,091] - [task:161] INFO -  {'uptime': '956', 'memoryTotal': 15466930176, 'memoryFree': 13145059328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-02 20:49:37,094] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-02 20:49:37,098] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-02 20:49:37,098] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-02 20:49:37,102] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-02 20:49:37,106] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:37,107] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:37,804] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:37,806] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:49:37,913] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:49:37,914] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:49:37,919] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-02 20:49:37,920] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-02 20:49:37,937] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-02 20:49:38,045] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:49:43,048] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-02 20:49:43,061] - [basetestcase:262] INFO - done initializing cluster
[2022-07-02 20:49:43,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:43,066] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:43,792] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:43,794] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-02 20:49:43,899] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-02 20:49:43,900] - [remote_util:5249] INFO - b'ok'
[2022-07-02 20:49:43,901] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-02 20:49:43,956] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-02 20:49:43,988] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-02 20:49:43,988] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-02 20:49:44,553] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:49:44,594] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:49:45,270] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-02 20:49:45,273] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-07-02 20:49:45,281] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:45,281] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:46,023] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:46,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:46,028] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:47,103] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:47,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:47,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:48,403] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:48,411] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:49:48,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:49:49,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:49:56,911] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:49:56,911] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.79881308581588, 'mem_free': 13142036480, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:49:56,911] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:49:56,911] - [basetestcase:467] INFO - Time to execute basesetup : 23.498239517211914
[2022-07-02 20:49:56,916] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:49:56,916] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:49:56,922] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:49:56,922] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:49:56,930] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:49:56,930] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:49:56,936] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-02 20:49:56,936] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-02 20:49:56,941] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:49:56,958] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-02 20:49:56,959] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-02 20:49:56,960] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-02 20:50:01,972] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-02 20:50:01,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:01,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:02,739] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:03,813] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-02 20:50:03,839] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-02 20:50:06,427] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-02 20:50:06,463] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:06,463] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-02 20:50:06,463] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-02 20:50:36,490] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-02 20:50:36,495] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:50:36,497] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:50:36,519] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.361536ms
[2022-07-02 20:50:36,520] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2d88abe2-788e-4351-876b-982f6c3ce277', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.361536ms', 'executionTime': '20.280356ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-02 20:50:36,520] - [tuq_helper:758] INFO - Create primary index
[2022-07-02 20:50:36,523] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-02 20:50:36,525] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-02 20:50:36,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 350.575351ms
[2022-07-02 20:50:36,878] - [tuq_helper:760] INFO - Check if index is online
[2022-07-02 20:50:36,885] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:50:36,892] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:50:36,907] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.746008ms
[2022-07-02 20:50:36,927] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:36,934] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-02 20:50:36,957] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-02 20:50:36,958] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-02 20:50:36,989] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-02 20:50:37,000] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:37,018] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:37,022] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:37,022] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:37,867] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:37,878] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:50:37,878] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-02 20:50:38,010] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:50:38,019] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:38,022] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:38,022] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:38,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:38,816] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-02 20:50:38,816] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-02 20:50:38,957] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-02 20:50:38,968] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-02 20:50:38,969] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-02 20:50:38,969] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:50:38,971] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-02 20:50:38,974] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-02 20:50:38,981] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.638828ms
[2022-07-02 20:50:38,983] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-02 20:50:38,985] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-02 20:50:39,029] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.884398ms
[2022-07-02 20:50:39,056] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-02 20:50:39,056] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.898016412742791, 'mem_free': 12972072960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-02 20:50:39,056] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-02 20:50:39,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:39,064] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:39,836] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:39,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:39,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:41,070] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:41,082] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:41,082] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:42,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:42,432] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-02 20:50:42,432] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-02 20:50:43,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-02 20:50:51,649] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-07-02 20:50:51,677] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-02 20:50:52,396] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-02 20:50:52,399] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-02 20:50:52,399] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-02 20:50:52,405] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:50:52,412] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:50:52,418] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-02 20:50:52,419] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-02 20:50:52,428] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-02 20:50:52,429] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-02 20:50:52,431] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:50:52,444] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-02 20:50:52,444] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:50:52,449] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-02 20:50:52,451] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-02 20:50:52,452] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:50:52,455] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-02 20:50:52,458] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-02 20:50:52,458] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:50:52,462] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-02 20:50:52,464] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-02 20:50:52,465] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-02 20:50:52,468] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-02 20:50:52,468] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-07-02 20:50:52,469] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-02 20:50:52,470] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 79.090s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 31438 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

Binary file /opt/build/test.log matches
Binary file /opt/build/test.log matches
Version: versions-02.07.2022-18.54.cfg
Build Log: make-02.07.2022-18.54.log
Server Log: logs-02.07.2022-18.54.tar.gz

Finished