Building

Started building at 2022/07/01 14:05:57
Using pegged server, 1432 build
Calculating base
Updating mirror
Basing run on 7.2.0-1432 c44d315bd
Updating tree for run 01.07.2022-14.05
query is at f979279, changes since last good build: none
gometa is at 8625ed1, changes since last good build: 

ns_server is at c9545cf, changes since last good build: none
couchstore is at f0c688c, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at d419a87, changes since last good build: none
Switching indexing to unstable
indexing is at b49b9d4, changes since last good build: none
Switching plasma to unstable
plasma is at 85217ad, changes since last good build: 
fatal: Invalid revision range 67b39ac64e16632744752cec29ead6c6bfd8d61e..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at a237ed7, changes since last good build: none
Switching testrunner to master
testrunner is at 6e16c8a, changes since last good build: none
Pulling in uncommitted change 175819 at refs/changes/19/175819/4
Total 5 (delta 2), reused 3 (delta 2)
[unstable ee20e148] MB-51825: kvsender - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 7 15:41:01 2022 +0530
 1 file changed, 42 insertions(+), 24 deletions(-)
Pulling in uncommitted change 176730 at refs/changes/30/176730/1
Total 5 (delta 4), reused 5 (delta 4)
[unstable 43fafcdf] MB-52151: Dont meter deletes in insert paths
 Author: Sai Krishna Teja Kommaraju 
 Date: Mon Jun 27 20:11:44 2022 +0530
 1 file changed, 11 insertions(+), 8 deletions(-)
Pulling in uncommitted change 176947 at refs/changes/47/176947/1
Total 20 (delta 17), reused 20 (delta 17)
[unstable 28137f06] MB-52308 Part 3 (7.2.0 1517) Elixir Pause-Resume feature for GSI
 Author: cherkauer-couchbase 
 Date: Thu Jun 30 17:33:44 2022 -0700
 9 files changed, 579 insertions(+), 335 deletions(-)
 create mode 100644 secondary/indexer/sched_token_monitor.go
Pulling in uncommitted change 176873 at refs/changes/73/176873/2
Total 3 (delta 0), reused 0 (delta 0)
[unstable e3561a6] MB-100: Fix addInstance for dedicated LSS cleanup
 Author: saptarshi.sen 
 Date: Wed Jun 29 11:07:35 2022 -0700
 1 file changed, 6 insertions(+)
Building cmakefiles and deps
Building main product

... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo


... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo

Testing

Started testing at 2022/07/01 14:38:11
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Jul 01 14:43:24 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 01 14:47:05 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 01 14:47:42 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Jul 01 14:49:02 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Jul 01 14:49:45 b" 'stop_on_failure': 'True'}"
Jul 01 14:49:45 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 01 14:49:45 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_1'}"
Jul 01 14:49:45 b'-->result: '
Jul 01 14:49:45 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Jul 01 14:49:45 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 01 14:49:45 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_2'}"
Jul 01 14:49:45 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 14:49:45 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Jul 01 14:49:45 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_3'}"
Jul 01 14:49:45 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 14:49:45 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 14:49:45 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Jul 01 14:49:45 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_4'}"
Jul 01 14:49:45 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 14:49:45 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 14:49:45 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 14:59:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Jul 01 15:00:32 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Jul 01 15:03:58 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 15:07:32 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 15:10:06 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 15:10:06 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_5'}"
Jul 01 15:10:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Jul 01 15:10:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Jul 01 15:10:06 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_6'}"
Jul 01 15:10:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 15:10:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Jul 01 15:10:06 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_7'}"
Jul 01 15:10:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Jul 01 15:10:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Jul 01 15:10:06 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_14-38-33/test_8'}"
Jul 01 15:10:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 15:10:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 15:10:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Jul 01 15:10:06 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Jul 01 15:10:06 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 01 15:10:06 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 01 15:10:06 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Jul 01 15:10:06 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Jul 01 15:10:06 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Jul 01 15:10:06 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Jul 01 15:10:06 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Jul 01 15:10:06 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.02s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000000 items took 26.650744119s -> 600358.4713641519 items/s conflicts 41
--- PASS: TestInsertPerf (26.65s)
=== RUN   TestGetPerf
16000000 items took 8.022695507s -> 1.9943421741532637e+06 items/s
--- PASS: TestGetPerf (8.62s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3351,
"memory_used":            37360992,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749144,
"level1": 187776,
"level2": 47222,
"level3": 11903,
"level4": 2974,
"level5": 735,
"level6": 182,
"level7": 48,
"level8": 12,
"level9": 2,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [35911 108875 357987 462344 598654 671258 781160]
No of items in each range [35911 72964 249112 104357 136310 72604 109902 218840]
--- PASS: TestGetRangeSplitItems (2.06s)
=== RUN   TestBuilder
{
"node_count":             50000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3333,
"memory_used":            1866650928,
"node_allocs":            50000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37500395,
"level1": 9374070,
"level2": 2344403,
"level3": 586130,
"level4": 146617,
"level5": 36338,
"level6": 9033,
"level7": 2299,
"level8": 539,
"level9": 123,
"level10": 39,
"level11": 9,
"level12": 5,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 7.171528083s to build 50000000 items, 6.972015e+06 items/sec
Took 434.852095ms to iterate 50000000 items
--- PASS: TestBuilder (7.61s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb/skiplist	45.190s
=== RUN   TestInsert
Set IO Concurrency: 716
--- PASS: TestInsert (0.01s)
=== RUN   TestInsertPerf
20000000 items took 51.041603027s -> 391837.22324356454 items/s snapshots_created 210 live_snapshots 1
--- PASS: TestInsertPerf (66.93s)
=== RUN   TestInsertDuplicates
--- PASS: TestInsertDuplicates (0.01s)
=== RUN   TestGetPerf
16000000 items took 11.717847537s -> 1.3654384859914565e+06 items/s
--- PASS: TestGetPerf (13.39s)
=== RUN   TestLoadStoreDisk
Inserting 1000000 items took 1.485752465s
{
"node_count":             999999,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       15,
"next_pointers_per_node": 1.3331,
"memory_used":            57328880,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749895,
"level1": 187701,
"level2": 46938,
"level3": 11656,
"level4": 2877,
"level5": 671,
"level6": 197,
"level7": 49,
"level8": 13,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
2022-07-01T15:12:40.705+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[59226 99347 62369 50660 39522 107258 77523 28533 37138 78308 38366 55058 14034 75314 48512 128832]] total count [1000000] snap count [1000000] size per shard [[473808 794776 498952 405280 316176 858064 620184 228264 297104 626464 306928 440464 112272 602512 388096 1030656]] total size [8000000] memoryInUse [57328880]
Storing to disk took 310.000633ms
Loading from disk took 290.834687ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3320,
"memory_used":            57311584,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750592,
"level1": 187126,
"level2": 46978,
"level3": 11552,
"level4": 2832,
"level5": 689,
"level6": 174,
"level7": 41,
"level8": 12,
"level9": 4,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestLoadStoreDisk (3.96s)
=== RUN   TestConcurrentLoadCloseFragmentation
Done Inserting 2000000 items
frag = NaN%
2022-07-01T15:12:46.482+05:30 [Info] MemDB::StoreToDisk: Unexpected amount of data being persisted dir [db.dump] count per shard [[32140 81354 244698 41179 114316 11785 192893 58119 156614 177113 170224 200941 173682 41179 62784 240979]] total count [2000000] snap count [1999999] size per shard [[257120 650832 1957584 329432 914528 94280 1543144 464952 1252912 1416904 1361792 1607528 1389456 329432 502272 1927832]] total size [16000000] memoryInUse [114667168]
2022-07-01T15:12:46.604+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[32140 81354 244698 41179 114316 11785 192893 58119 156614 177113 170224 200941 173682 41179 62784 240979]] total count [2000000] snap count [1999999] size per shard [[257120 650832 1957584 329432 914528 94280 1543144 464952 1252912 1416904 1361792 1607528 1389456 329432 502272 1927832]] total size [16000000] memoryInUse [114667168]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentation (5.90s)
=== RUN   TestConcurrentLoadCloseFragmentationSmall
Done Inserting 10 items
frag = NaN%
2022-07-01T15:12:48.596+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationSmall (0.06s)
=== RUN   TestConcurrentLoadCloseFragmentationEmpty
Done Inserting 0 items
frag = NaN%
2022-07-01T15:12:48.653+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationEmpty (0.40s)
=== RUN   TestConcurrentCloseSingleNode
A
--- PASS: TestConcurrentCloseSingleNode (0.00s)
=== RUN   TestStoreDiskShutdown
Inserting 1000000 items took 1.478203028s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       8,
"next_pointers_per_node": 1.3346,
"memory_used":            57352944,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749112,
"level1": 188260,
"level2": 46826,
"level3": 11850,
"level4": 2973,
"level5": 740,
"level6": 181,
"level7": 45,
"level8": 13,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
Prepare returns error:  MemDB instance has been shutdown
--- PASS: TestStoreDiskShutdown (2.32s)
=== RUN   TestDelete
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.1000,
"memory_used":            556,
"node_allocs":            10,
"node_frees":             0,
"level_node_distribution":{
"level0": 9,
"level1": 1,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.2000,
"memory_used":            572,
"node_allocs":            20,
"node_frees":             10,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestDelete (1.00s)
=== RUN   TestGCPerf
final_node_count = 16000, average_live_node_count = 67919, wait_time_for_collection = 15ms
--- PASS: TestGCPerf (35.98s)
=== RUN   TestMemoryInUse
ItemsCount: 5000, MemoryInUse: 297264, NodesCount: 5000
ItemsCount: 0, MemoryInUse: 64, NodesCount: 0
--- PASS: TestMemoryInUse (1.01s)
=== RUN   TestFullScan
Inserting 1600000 items took 2.484706179s
{
"node_count":             1600000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       7,
"next_pointers_per_node": 1.3339,
"memory_used":            91748128,
"node_allocs":            1600000,
"node_frees":             0,
"level_node_distribution":{
"level0": 1199920,
"level1": 299458,
"level2": 75450,
"level3": 18893,
"level4": 4710,
"level5": 1166,
"level6": 298,
"level7": 81,
"level8": 21,
"level9": 2,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Full iteration of 1600000 items took 435.998196ms
--- PASS: TestFullScan (4.22s)
=== RUN   TestVisitor
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3334,
"memory_used":            57334000,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750167,
"level1": 187306,
"level2": 46757,
"level3": 11807,
"level4": 3008,
"level5": 719,
"level6": 168,
"level7": 50,
"level8": 13,
"level9": 5,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 43.579406ms to iterate 1000000 items, 2.2946618e+07 items/s
shard - 0 count = 15476, range: 0-15475
shard - 1 count = 31791, range: 15476-47266
shard - 2 count = 19400, range: 47267-66666
shard - 3 count = 52107, range: 66667-118773
shard - 4 count = 8838, range: 118774-127611
shard - 5 count = 14597, range: 127612-142208
shard - 6 count = 25927, range: 142209-168135
shard - 7 count = 29252, range: 168136-197387
shard - 8 count = 22438, range: 197388-219825
shard - 9 count = 13712, range: 219826-233537
shard - 10 count = 46242, range: 233538-279779
shard - 11 count = 26197, range: 279780-305976
shard - 12 count = 10203, range: 305977-316179
shard - 13 count = 9399, range: 316180-325578
shard - 14 count = 32805, range: 325579-358383
shard - 15 count = 21092, range: 358384-379475
shard - 16 count = 75045, range: 379476-454520
shard - 17 count = 41986, range: 454521-496506
shard - 18 count = 21241, range: 496507-517747
shard - 19 count = 45296, range: 517748-563043
shard - 20 count = 110793, range: 563044-673836
shard - 21 count = 8465, range: 673837-682301
shard - 22 count = 16677, range: 682302-698978
shard - 23 count = 22027, range: 698979-721005
shard - 24 count = 22558, range: 721006-743563
shard - 25 count = 35851, range: 743564-779414
shard - 26 count = 17083, range: 779415-796497
shard - 27 count = 8032, range: 796498-804529
shard - 28 count = 22190, range: 804530-826719
shard - 29 count = 1960, range: 826720-828679
shard - 30 count = 25847, range: 828680-854526
shard - 31 count = 145473, range: 854527-999999
--- PASS: TestVisitor (1.44s)
=== RUN   TestVisitorError
--- PASS: TestVisitorError (0.15s)
=== RUN   TestLoadDeltaStoreDisk
snap count = 1000000
2022-07-01T15:13:39.196+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 32360 120047 13262 178537 267009 162026 70347 131751 0]] total count [975339] snap count [1000000] size per shard [[0 0 0 0 0 0 0 258880 960376 106096 1428296 2136072 1296208 562776 1054008 0]] total size [7802712] memoryInUse [130326816]
Storing to disk took 15.364227145s
Loading from disk took 751.282436ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3335,
"memory_used":            57335824,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749903,
"level1": 187478,
"level2": 47074,
"level3": 11602,
"level4": 2974,
"level5": 731,
"level6": 176,
"level7": 51,
"level8": 7,
"level9": 3,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Restored 24661
RestoredFailed 407272
--- PASS: TestLoadDeltaStoreDisk (18.26s)
=== RUN   TestExecuteConcurrGCWorkers
--- PASS: TestExecuteConcurrGCWorkers (0.55s)
=== RUN   TestCloseWithActiveIterators
--- PASS: TestCloseWithActiveIterators (37.52s)
=== RUN   TestDiskCorruption
Inserting 100000 items took 98.191268ms
{
"node_count":             100000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       1,
"next_pointers_per_node": 1.3310,
"memory_used":            5729552,
"node_allocs":            100000,
"node_frees":             0,
"level_node_distribution":{
"level0": 75139,
"level1": 18669,
"level2": 4660,
"level3": 1147,
"level4": 294,
"level5": 63,
"level6": 22,
"level7": 4,
"level8": 2,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
2022-07-01T15:14:31.636+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[719 4600 4164 5591 9799 5397 4718 8042 4712 1239 9009 6951 6428 12357 16274 0]] total count [100000] snap count [100000] size per shard [[5752 36800 33312 44728 78392 43176 37744 64336 37696 9912 72072 55608 51424 98856 130192 0]] total size [800000] memoryInUse [5729552]
Storing to disk took 72.235188ms
Loading from disk took 34.307711ms
--- PASS: TestDiskCorruption (0.72s)
=== RUN   TestSnapshotStats
--- PASS: TestSnapshotStats (0.00s)
=== RUN   TestInsertDeleteConcurrent
total items in snapshot: 7500000  items deleted: 2500000
{
"node_count":             7500000,
"soft_deletes":           0,
"read_conflicts":         6369,
"insert_conflicts":       187,
"next_pointers_per_node": 1.3332,
"memory_used":            429978736,
"node_allocs":            10000000,
"node_frees":             10000002,
"level_node_distribution":{
"level0": 5625606,
"level1": 1406334,
"level2": 350891,
"level3": 87858,
"level4": 22018,
"level5": 5465,
"level6": 1371,
"level7": 338,
"level8": 87,
"level9": 26,
"level10": 4,
"level11": 2,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
allocs: 20000002 frees: 20000002
--- PASS: TestInsertDeleteConcurrent (21.21s)
=== RUN   TestNodeList
--- PASS: TestNodeList (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb	215.325s
=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.035s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-07-01T15:15:05.403+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5473 us

2022-07-01T15:15:05.404+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-01T15:15:05.408+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-07-01T15:15:05.408+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-01T15:15:05.411+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 1m50.156463397s -> 90779.96598311579 items/s
Incr build: 10000000 items took 1m57.170738499s -> 85345.54043188306 items/s
Main Index: {
"node_count":             18000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       2,
"next_pointers_per_node": 1.3333,
"memory_used":            1695887644,
"node_allocs":            18000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 13500190,
"level1": 3375549,
"level2": 843075,
"level3": 210794,
"level4": 52734,
"level5": 13348,
"level6": 3245,
"level7": 782,
"level8": 220,
"level9": 42,
"level10": 17,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (227.33s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	251.410s
=== RUN   TestConnPoolBasicSanity
2022-07-01T15:19:21.096+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-07-01T15:19:21.303+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:19:22.096+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-07-01T15:19:26.099+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-07-01T15:20:05.858+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:20:06.116+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.76s)
=== RUN   TestLongevity
2022-07-01T15:20:09.860+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-07-01T15:20:48.340+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:20:48.880+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.48s)
=== RUN   TestSustainedHighConns
2022-07-01T15:20:52.341+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Allocating 16 Connections
cp.curActConns = 0
Returning 3 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 14
Returning 1 Connections
Allocating 12 Connections
cp.curActConns = 23
Returning 1 Connections
cp.curActConns = 24
Allocating 10 Connections
Returning 1 Connections
cp.curActConns = 33
Returning 3 Connections
Allocating 15 Connections
cp.curActConns = 34
Returning 4 Connections
cp.curActConns = 41
Returning 3 Connections
Allocating 8 Connections
cp.curActConns = 46
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 46
Returning 4 Connections
cp.curActConns = 43
Allocating 9 Connections
Returning 3 Connections
cp.curActConns = 49
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 57
Returning 4 Connections
cp.curActConns = 64
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 60
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 68
Returning 3 Connections
cp.curActConns = 70
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 72
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 79
Returning 0 Connections
cp.curActConns = 81
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 91
Returning 3 Connections
cp.curActConns = 93
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 94
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 102
Returning 3 Connections
cp.curActConns = 109
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 110
Returning 3 Connections
Allocating 22 Connections
cp.curActConns = 114
Returning 4 Connections
cp.curActConns = 125
Returning 2 Connections
Allocating 13 Connections
cp.curActConns = 125
Returning 1 Connections
cp.curActConns = 135
Returning 0 Connections
Allocating 23 Connections
cp.curActConns = 136
Returning 3 Connections
cp.curActConns = 147
Returning 2 Connections
cp.curActConns = 153
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 158
Returning 4 Connections
cp.curActConns = 162
Returning 3 Connections
Allocating 18 Connections
cp.curActConns = 169
Returning 1 Connections
cp.curActConns = 176
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 177
Returning 1 Connections
Allocating 21 Connections
cp.curActConns = 180
Returning 2 Connections
cp.curActConns = 191
Returning 0 Connections
cp.curActConns = 195
Returning 3 Connections
Allocating 2 Connections
cp.curActConns = 194
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 196
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 193
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 203
Returning 2 Connections
cp.curActConns = 205
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 203
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 203
Returning 0 Connections
Allocating 2 Connections
cp.curActConns = 205
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 205
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 204
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 204
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 207
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 210
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 208
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 205
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 203
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 203
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 205
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 205
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 204
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 202
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 202
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 205
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 206
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 207
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 210
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 210
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 211
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 212
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 211
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 210
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 213
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 0 Connections
cp.curActConns = 217
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 212
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 212
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 217
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 215
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 218
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 217
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 220
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 220
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-07-01T15:21:47.408+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:21:48.377+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.07s)
=== RUN   TestLowWM
2022-07-01T15:21:51.409+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-07-01T15:22:51.425+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-01T15:23:51.441+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-01T15:23:56.922+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:23:57.442+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN   TestTotalConns
2022-07-01T15:24:00.924+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-07-01T15:24:15.110+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:24:15.930+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.19s)
=== RUN   TestUpdateTickRate
2022-07-01T15:24:19.111+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-07-01T15:24:39.959+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T15:24:40.120+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	322.915s
Starting server: attempt 1

Functional tests

# github.com/couchbase/indexing/secondary/tests/framework/secondaryindex
../framework/secondaryindex/n1qlclient.go:715:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:719:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:723:47: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:727:47: undefined: tenant.Unit
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests [build failed]
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
# github.com/couchbase/indexing/secondary/tests/framework/secondaryindex
../framework/secondaryindex/n1qlclient.go:715:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:719:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:723:47: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:727:47: undefined: tenant.Unit
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests [build failed]
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.16.6
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.16.6
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> chronicle (compile)
[100%] Built target chronicle
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target golang_tls_test using Go 1.17.4
[100%] Built target golang_tls_test
[100%] Building Go target ns_goport using Go 1.18.1
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.1
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.1
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.1
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.1
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-07-01 15:27:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:31 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 15:27:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 15:27:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-01 15:27:57 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:57 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-07-01 15:27:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5584748544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:27:58 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-01 15:27:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:27:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5408292864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:27:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:27:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:28:00 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 5288538112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:28:00 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:28:00 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 5183750144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:28:00 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:28:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:28:00 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-01 15:28:01 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 15:28:01 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-07-01 15:28:01 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:28:06 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-01 15:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-01 15:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.03 seconds to create bucket default
2022-07-01 15:28:07 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-01 15:28:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:28:07 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:28:08 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 27.0278178202827, 'mem_free': 13548883968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 40.44474816322327
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-01 15:28:11 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 15:28:16 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-01 15:28:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:17 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-07-01 15:28:17 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:28:20 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-07-01 15:28:20 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:20 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-01 15:28:20 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 100.363885ms
2022-07-01 15:28:50 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '14739175-ecae-46c4-9dab-23c775008359', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '100.363885ms', 'executionTime': '100.195687ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 464.340586ms
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 13.387148ms
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.851288ms
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 50.04535ms
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.487928406584071, 'mem_free': 13525811200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 15:28:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 15:28:56 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.360s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.487928406584071, 'mem_free': 13525811200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:28:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:28:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:28:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 15:29:02 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-01 15:29:02 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '88', 'memoryTotal': 15466930176, 'memoryFree': 13502431232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-01 15:29:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '87', 'memoryTotal': 15466930176, 'memoryFree': 13498060800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '83', 'memoryTotal': 15466930176, 'memoryFree': 13477953536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '82', 'memoryTotal': 15466930176, 'memoryFree': 13505667072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 15:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-07-01 15:29:04 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-01 15:29:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 15:29:09 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-01 15:29:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-01 15:29:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.03 seconds to create bucket default
2022-07-01 15:29:10 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-01 15:29:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:29:11 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:29:11 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-01 15:29:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 15:29:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:11 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:12 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.06019279910939, 'mem_free': 13488209920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 19.12067198753357
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-01 15:29:15 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 15:29:20 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-01 15:29:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:29:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:29:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:29:21 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-07-01 15:29:22 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 15:29:24 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-07-01 15:29:24 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:29:24 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-01 15:29:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 19.812986ms
2022-07-01 15:29:54 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '781b4892-b239-45c5-84c6-e7c220571612', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.812986ms', 'executionTime': '19.67047ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 385.414846ms
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.728676ms
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-01 15:29:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:29:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee50d5474971394b19a1c89e7bde215b44job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-07-01 15:29:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee50d5474971394b19a1c89e7bde215b44job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-07-01 15:29:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 45.894769ms
2022-07-01 15:29:55 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee50d5474971394b19a1c89e7bde215b44job_title) USING GSI
2022-07-01 15:29:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee50d5474971394b19a1c89e7bde215b44job_title) USING GSI
2022-07-01 15:29:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee50d5474971394b19a1c89e7bde215b44job_title%29+USING+GSI
2022-07-01 15:29:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.37683ms
2022-07-01 15:29:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee50d5474971394b19a1c89e7bde215b44job_title'
2022-07-01 15:29:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee50d5474971394b19a1c89e7bde215b44job_title%27
2022-07-01 15:29:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.718284ms
2022-07-01 15:29:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee50d5474971394b19a1c89e7bde215b44job_title'
2022-07-01 15:29:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee50d5474971394b19a1c89e7bde215b44job_title%27
2022-07-01 15:29:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.716876ms
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.140172ms
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '4f8ad298-3473-4f4b-9992-8900f12d799f', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee50d5474971394b19a1c89e7bde215b44job_title', 'index_id': 'd598d023013db3d5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.140172ms', 'executionTime': '3.009963ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:29:59 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-07-01 15:29:59 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 82.98523ms
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-07-01 15:30:00 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-07-01 15:30:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:30:01 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee50d5474971394b19a1c89e7bde215b44job_title'
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee50d5474971394b19a1c89e7bde215b44job_title%27
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.49552ms
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee50d5474971394b19a1c89e7bde215b44job_title ON default USING GSI
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee50d5474971394b19a1c89e7bde215b44job_title+ON+default+USING+GSI
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 54.136714ms
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee50d5474971394b19a1c89e7bde215b44job_title'
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee50d5474971394b19a1c89e7bde215b44job_title%27
2022-07-01 15:30:02 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.645992ms
2022-07-01 15:30:02 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '94fa36fe-fd68-46ab-8d18-e28e83d7b545', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.645992ms', 'executionTime': '1.565334ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 15:30:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:30:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.165058ms
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 42.933823ms
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.23086540343035, 'mem_free': 13377585152, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 15:30:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 15:30:09 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 15:30:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 73.229s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_2'}
[2022-07-01 15:30:09,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:09,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:10,251] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:10,257] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:30:10,266] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:30:10,267] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-07-01 15:30:10,267] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:30:10,936] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:30:10,940] - [task:161] INFO -  {'uptime': '156', 'memoryTotal': 15466930176, 'memoryFree': 13420412928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:30:10,942] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:30:10,942] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:30:10,942] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:30:10,951] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:30:10,959] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:30:10,960] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:30:10,962] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:30:10,964] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:30:10,964] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:30:10,964] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:30:10,967] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:30:10,970] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:10,970] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:11,243] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:11,244] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:30:11,290] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:30:11,291] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:30:11,296] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:30:11,296] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:30:11,325] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:30:11,329] - [task:161] INFO -  {'uptime': '153', 'memoryTotal': 15466930176, 'memoryFree': 13382791168, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:30:11,331] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:30:11,335] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:30:11,336] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:30:11,338] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:30:11,342] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:11,342] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:11,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:11,607] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:30:11,650] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:30:11,650] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:30:11,655] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:30:11,656] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:30:11,682] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:30:11,685] - [task:161] INFO -  {'uptime': '153', 'memoryTotal': 15466930176, 'memoryFree': 13386096640, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:30:11,687] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:30:11,691] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:30:11,692] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:30:11,694] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:30:11,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:11,701] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:11,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:11,972] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:30:12,021] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:30:12,022] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:30:12,027] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:30:12,028] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:30:12,049] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:30:12,053] - [task:161] INFO -  {'uptime': '148', 'memoryTotal': 15466930176, 'memoryFree': 13387935744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:30:12,055] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:30:12,060] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:30:12,061] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:30:12,064] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:30:12,068] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:12,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:12,355] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:12,356] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:30:12,400] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:30:12,401] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:30:12,406] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:30:12,406] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:30:12,422] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:30:12,529] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:30:17,535] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:30:17,546] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:30:17,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:17,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:17,819] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:17,820] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:30:17,864] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:30:17,865] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:30:17,865] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:30:18,439] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:30:18,470] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:30:18,470] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:30:19,032] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:30:19,072] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:30:19,747] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:30:19,750] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:30:19,761] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:19,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:20,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:20,068] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:20,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:20,317] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:20,322] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:20,322] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:20,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:20,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:20,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:20,923] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:24,454] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:30:24,455] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.62705563527818, 'mem_free': 13417746432, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:30:24,455] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:30:24,455] - [basetestcase:467] INFO - Time to execute basesetup : 14.526399850845337
[2022-07-01 15:30:24,459] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:30:24,460] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:30:24,466] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:30:24,466] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:30:24,472] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:30:24,472] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:30:24,478] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:30:24,478] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:30:24,483] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:30:24,496] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:30:24,496] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:30:24,496] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:30:29,508] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:30:29,513] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:30:29,513] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:30:29,769] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:30:30,727] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:30:30,752] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:30:32,851] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:30:32,868] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 15:30:33,496] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:30:33,496] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:30:33,496] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:31:03,503] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:31:03,508] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:31:03,510] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:31:03,531] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.535544ms
[2022-07-01 15:31:03,532] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd3a0c7c1-3739-401e-8504-fb45d3b5904c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.535544ms', 'executionTime': '19.452019ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:31:03,532] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:31:03,534] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:31:03,536] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:31:03,907] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 368.432644ms
[2022-07-01 15:31:03,907] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:31:03,915] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:31:03,922] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:31:03,936] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 11.43118ms
[2022-07-01 15:31:03,983] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:03,987] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:31:04,002] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:31:04,002] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:31:04,016] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:31:04,026] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:04,980] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee00090ed7a94e43b5989e33963b88b2aejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:31:04,982] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee00090ed7a94e43b5989e33963b88b2aejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:31:05,016] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.040355ms
[2022-07-01 15:31:05,016] - [base_gsi:282] INFO - BUILD INDEX on default(employee00090ed7a94e43b5989e33963b88b2aejob_title) USING GSI
[2022-07-01 15:31:06,022] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee00090ed7a94e43b5989e33963b88b2aejob_title) USING GSI
[2022-07-01 15:31:06,024] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee00090ed7a94e43b5989e33963b88b2aejob_title%29+USING+GSI
[2022-07-01 15:31:06,055] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.004016ms
[2022-07-01 15:31:07,061] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00090ed7a94e43b5989e33963b88b2aejob_title'
[2022-07-01 15:31:07,064] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00090ed7a94e43b5989e33963b88b2aejob_title%27
[2022-07-01 15:31:07,073] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.859602ms
[2022-07-01 15:31:08,080] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00090ed7a94e43b5989e33963b88b2aejob_title'
[2022-07-01 15:31:08,082] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00090ed7a94e43b5989e33963b88b2aejob_title%27
[2022-07-01 15:31:08,091] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.5593ms
[2022-07-01 15:31:08,568] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-01 15:31:08,590] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:31:09,566] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:31:10,094] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:10,102] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:31:10,104] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 15:31:10,108] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.475761ms
[2022-07-01 15:31:10,108] - [task:3216] INFO - {'requestID': 'c4a33845-e68e-4643-b6e1-67f87fee9d51', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee00090ed7a94e43b5989e33963b88b2aejob_title', 'index_id': '71976760e311e7d2', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.475761ms', 'executionTime': '2.341644ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 15:31:10,108] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:10,109] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:10,109] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:31:10,109] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:31:10,110] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 15:31:10,110] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:31:10,110] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:31:10,110] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:31:10,110] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:31:11,110] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:11,116] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:31:11,118] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 15:31:11,197] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.480125ms
[2022-07-01 15:31:11,197] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:31:11,197] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:31:11,950] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:11,951] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:31:12,958] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00090ed7a94e43b5989e33963b88b2aejob_title'
[2022-07-01 15:31:12,960] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00090ed7a94e43b5989e33963b88b2aejob_title%27
[2022-07-01 15:31:12,970] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.878881ms
[2022-07-01 15:31:12,972] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee00090ed7a94e43b5989e33963b88b2aejob_title ON default USING GSI
[2022-07-01 15:31:12,974] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee00090ed7a94e43b5989e33963b88b2aejob_title+ON+default+USING+GSI
[2022-07-01 15:31:13,019] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.301946ms
[2022-07-01 15:31:13,026] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00090ed7a94e43b5989e33963b88b2aejob_title'
[2022-07-01 15:31:13,028] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00090ed7a94e43b5989e33963b88b2aejob_title%27
[2022-07-01 15:31:13,032] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.533507ms
[2022-07-01 15:31:13,032] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '62ea66da-3996-4e56-bb57-fc912c7f1183', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.533507ms', 'executionTime': '1.472386ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:31:13,041] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:13,044] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:13,044] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:13,418] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:13,425] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:31:13,426] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:31:13,489] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:31:13,497] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:13,501] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:13,501] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:13,865] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:13,873] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:31:13,873] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:31:13,934] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:31:13,942] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:13,943] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:31:13,943] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:31:13,945] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:31:13,949] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:31:13,957] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.249041ms
[2022-07-01 15:31:13,959] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:31:13,961] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:31:14,017] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.662747ms
[2022-07-01 15:31:14,028] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:31:14,028] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.2120152851231, 'mem_free': 13289680896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:31:14,029] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:31:14,033] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:14,033] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:14,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:14,387] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:14,387] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:14,836] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:14,843] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:14,843] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:15,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:15,446] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:15,446] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:16,062] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:19,909] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:31:19,934] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:31:20,812] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:31:20,815] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:31:20,815] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:31:20,821] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:31:20,828] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:31:20,833] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:31:20,835] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:31:20,843] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:31:20,844] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:31:20,846] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:20,860] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:31:20,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:20,864] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:31:20,867] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:31:20,868] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:20,871] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:31:20,874] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:31:20,874] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:20,877] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:31:20,879] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:31:20,879] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:20,882] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:31:20,882] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:31:20,883] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:31:20,883] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 70.960s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_3'}
[2022-07-01 15:31:20,935] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:20,936] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:21,260] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:21,266] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:31:21,274] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:31:21,274] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-07-01 15:31:21,275] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:31:21,940] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:31:21,943] - [task:161] INFO -  {'uptime': '227', 'memoryTotal': 15466930176, 'memoryFree': 13364412416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:31:21,945] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:31:21,945] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:31:21,946] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:31:21,958] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:31:21,966] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:31:21,967] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:31:21,969] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:31:21,970] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:31:21,971] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:31:21,971] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:31:21,973] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:31:21,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:21,976] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:22,291] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:22,292] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:31:22,344] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:31:22,345] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:31:22,349] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:31:22,350] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:31:22,382] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:31:22,386] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13291765760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:31:22,388] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:31:22,396] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:31:22,396] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:31:22,398] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:31:22,401] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:22,401] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:22,718] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:22,719] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:31:22,781] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:31:22,782] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:31:22,787] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:31:22,788] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:31:22,808] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:31:22,811] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13291765760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:31:22,813] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:31:22,820] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:31:22,820] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:31:22,822] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:31:22,828] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:22,828] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:23,145] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:23,146] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:31:23,196] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:31:23,197] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:31:23,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:31:23,203] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:31:23,222] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:31:23,225] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13288177664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:31:23,227] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:31:23,233] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:31:23,233] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:31:23,235] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:31:23,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:23,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:23,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:23,554] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:31:23,602] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:31:23,603] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:31:23,608] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:31:23,610] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:31:23,633] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:31:23,778] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:31:28,783] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:31:28,796] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:31:28,801] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:28,801] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:29,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:29,117] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:31:29,170] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:31:29,171] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:31:29,172] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:31:29,650] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:31:29,679] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:31:29,680] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:31:30,243] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:31:30,282] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:31:30,969] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:31:30,972] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:31:30,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:30,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:31,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:31,387] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:31,387] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:31,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:31,698] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:31,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:32,042] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:32,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:32,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:32,602] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:36,747] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:31:36,747] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.0685356863018, 'mem_free': 13368446976, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:31:36,747] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:31:36,748] - [basetestcase:467] INFO - Time to execute basesetup : 15.814799785614014
[2022-07-01 15:31:36,752] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:31:36,752] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:31:36,759] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:31:36,759] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:31:36,765] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:31:36,765] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:31:36,771] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:31:36,771] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:31:36,775] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:36,790] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:31:36,791] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:31:36,791] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:31:41,803] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:31:41,807] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:31:41,808] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:31:42,123] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:31:43,050] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:31:43,073] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:31:45,146] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:31:45,171] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 15:31:45,950] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:31:45,951] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:31:45,951] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:32:15,972] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:32:15,978] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:32:15,980] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:32:16,001] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.191823ms
[2022-07-01 15:32:16,001] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7c656485-f89d-45a5-a6fa-aa29c4d309f3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.191823ms', 'executionTime': '19.111068ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:32:16,001] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:32:16,003] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:32:16,005] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:32:16,386] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 378.917583ms
[2022-07-01 15:32:16,386] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:32:16,399] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:32:16,403] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:32:16,413] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.405327ms
[2022-07-01 15:32:16,450] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:16,454] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:32:16,472] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:32:16,473] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:32:16,490] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:32:16,504] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:17,446] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb20eff8d947c424aa4a0f07ef6ded250job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:32:17,449] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb20eff8d947c424aa4a0f07ef6ded250job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:32:17,480] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.164117ms
[2022-07-01 15:32:17,480] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb20eff8d947c424aa4a0f07ef6ded250job_title) USING GSI
[2022-07-01 15:32:18,485] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb20eff8d947c424aa4a0f07ef6ded250job_title) USING GSI
[2022-07-01 15:32:18,487] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb20eff8d947c424aa4a0f07ef6ded250job_title%29+USING+GSI
[2022-07-01 15:32:18,509] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.439082ms
[2022-07-01 15:32:19,516] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb20eff8d947c424aa4a0f07ef6ded250job_title'
[2022-07-01 15:32:19,518] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb20eff8d947c424aa4a0f07ef6ded250job_title%27
[2022-07-01 15:32:19,527] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.581095ms
[2022-07-01 15:32:20,534] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb20eff8d947c424aa4a0f07ef6ded250job_title'
[2022-07-01 15:32:20,536] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb20eff8d947c424aa4a0f07ef6ded250job_title%27
[2022-07-01 15:32:20,546] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.424108ms
[2022-07-01 15:32:21,036] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 15:32:21,059] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:32:21,890] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:32:22,549] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:22,557] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:32:22,559] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 15:32:22,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.142713ms
[2022-07-01 15:32:22,564] - [task:3216] INFO - {'requestID': 'e8e2a6c7-7172-4e4d-b88b-4dad1ef72094', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb20eff8d947c424aa4a0f07ef6ded250job_title', 'index_id': '5784bbcf064973d4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.142713ms', 'executionTime': '3.059268ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 15:32:22,565] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:22,565] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:22,565] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:32:22,566] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:32:22,566] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 15:32:22,566] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:32:22,567] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:32:22,567] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:32:22,567] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:32:23,567] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:23,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:32:23,574] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 15:32:23,645] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.8062ms
[2022-07-01 15:32:23,645] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:32:23,645] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:32:24,591] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:24,591] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:32:25,598] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb20eff8d947c424aa4a0f07ef6ded250job_title'
[2022-07-01 15:32:25,600] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb20eff8d947c424aa4a0f07ef6ded250job_title%27
[2022-07-01 15:32:25,610] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.217472ms
[2022-07-01 15:32:25,612] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb20eff8d947c424aa4a0f07ef6ded250job_title ON default USING GSI
[2022-07-01 15:32:25,614] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb20eff8d947c424aa4a0f07ef6ded250job_title+ON+default+USING+GSI
[2022-07-01 15:32:25,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.982226ms
[2022-07-01 15:32:25,676] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb20eff8d947c424aa4a0f07ef6ded250job_title'
[2022-07-01 15:32:25,679] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb20eff8d947c424aa4a0f07ef6ded250job_title%27
[2022-07-01 15:32:25,682] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.563769ms
[2022-07-01 15:32:25,682] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '96b8471d-f290-48c4-ad35-2ca93e598f7d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.563769ms', 'executionTime': '1.500156ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:32:25,691] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:25,693] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:25,693] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:26,132] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:26,140] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:32:26,141] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:32:26,208] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:32:26,218] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:26,221] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:26,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:26,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:26,633] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:32:26,633] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:32:26,701] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:32:26,711] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:26,711] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:32:26,711] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:32:26,713] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:32:26,716] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:32:26,725] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.947276ms
[2022-07-01 15:32:26,727] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:32:26,729] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:32:26,766] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.394892ms
[2022-07-01 15:32:26,797] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:32:26,797] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 9.079414677662644, 'mem_free': 13236248576, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:32:26,797] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:32:26,802] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:26,802] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:27,225] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:27,230] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:27,230] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:27,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:27,653] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:27,654] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:28,090] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:28,150] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:28,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:28,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:33,865] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:32:33,888] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:32:35,083] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:32:35,086] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:32:35,086] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:32:35,092] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:32:35,098] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:32:35,104] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:32:35,105] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:32:35,114] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:32:35,115] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:32:35,118] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,130] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:32:35,130] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,134] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:32:35,136] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:32:35,136] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,139] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:32:35,141] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:32:35,141] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,145] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:32:35,147] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:32:35,147] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,150] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:32:35,150] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:32:35,151] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:32:35,151] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 74.231s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_4'}
[2022-07-01 15:32:35,214] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:35,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:35,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:35,601] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:32:35,611] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:32:35,611] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-07-01 15:32:35,612] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:32:36,219] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:32:36,222] - [task:161] INFO -  {'uptime': '301', 'memoryTotal': 15466930176, 'memoryFree': 13215313920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:32:36,224] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:32:36,224] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:32:36,225] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:32:36,230] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:32:36,236] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:32:36,237] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:32:36,240] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:32:36,241] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:32:36,242] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:32:36,242] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:32:36,245] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:32:36,250] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:36,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:36,636] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:36,637] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:32:36,696] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:32:36,698] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:32:36,703] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:32:36,704] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:32:36,730] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:32:36,734] - [task:161] INFO -  {'uptime': '298', 'memoryTotal': 15466930176, 'memoryFree': 13307117568, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:32:36,736] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:32:36,739] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:32:36,740] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:32:36,742] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:32:36,746] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:36,746] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:37,152] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:37,153] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:32:37,216] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:32:37,217] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:32:37,222] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:32:37,223] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:32:37,242] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:32:37,246] - [task:161] INFO -  {'uptime': '299', 'memoryTotal': 15466930176, 'memoryFree': 13307113472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:32:37,248] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:32:37,255] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:32:37,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:32:37,259] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:32:37,265] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:37,266] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:37,643] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:37,644] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:32:37,705] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:32:37,706] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:32:37,711] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:32:37,712] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:32:37,739] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:32:37,742] - [task:161] INFO -  {'uptime': '293', 'memoryTotal': 15466930176, 'memoryFree': 13235814400, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:32:37,744] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:32:37,749] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:32:37,749] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:32:37,751] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:32:37,756] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:37,756] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:38,142] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:38,143] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:32:38,202] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:32:38,204] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:32:38,209] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:32:38,210] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:32:38,229] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:32:38,337] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:32:43,343] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:32:43,353] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:32:43,359] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:43,359] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:43,751] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:43,752] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:32:43,813] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:32:43,814] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:32:43,814] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:32:44,247] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:32:44,278] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:32:44,278] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:32:44,840] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:32:44,879] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:32:45,556] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:32:45,558] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:32:45,566] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:45,568] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:45,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:45,994] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:45,995] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:46,366] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:46,371] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:46,371] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:46,901] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:46,912] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:46,912] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:47,567] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:52,064] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:32:52,064] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.10108418341836, 'mem_free': 13307064320, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:32:52,065] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:32:52,065] - [basetestcase:467] INFO - Time to execute basesetup : 16.852829217910767
[2022-07-01 15:32:52,069] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:32:52,070] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:32:52,076] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:32:52,077] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:32:52,083] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:32:52,083] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:32:52,089] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:32:52,089] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:32:52,093] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:32:52,108] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:32:52,108] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:32:52,109] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:32:57,116] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:32:57,121] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:32:57,121] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:32:57,499] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:32:58,504] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:32:58,530] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:33:00,536] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:33:00,553] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 15:33:01,219] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:01,219] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:33:01,219] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:33:31,234] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:33:31,240] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:33:31,242] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:33:31,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.808333ms
[2022-07-01 15:33:31,262] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1d705f25-d735-44d4-8587-1e9678114e8f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '17.808333ms', 'executionTime': '17.740115ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:33:31,262] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:33:31,264] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:33:31,266] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:33:31,639] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 371.045441ms
[2022-07-01 15:33:31,639] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:33:31,647] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:33:31,655] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:33:31,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.09603ms
[2022-07-01 15:33:31,697] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:31,704] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:33:31,721] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:33:31,722] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:33:31,734] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:33:31,747] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:32,697] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8647b7e037124214af5d243cca5761b6job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:33:32,699] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8647b7e037124214af5d243cca5761b6job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:33:32,744] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.218789ms
[2022-07-01 15:33:32,745] - [base_gsi:282] INFO - BUILD INDEX on default(employee8647b7e037124214af5d243cca5761b6job_title) USING GSI
[2022-07-01 15:33:33,751] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee8647b7e037124214af5d243cca5761b6job_title) USING GSI
[2022-07-01 15:33:33,753] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee8647b7e037124214af5d243cca5761b6job_title%29+USING+GSI
[2022-07-01 15:33:33,783] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.491971ms
[2022-07-01 15:33:34,789] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8647b7e037124214af5d243cca5761b6job_title'
[2022-07-01 15:33:34,791] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8647b7e037124214af5d243cca5761b6job_title%27
[2022-07-01 15:33:34,800] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.004839ms
[2022-07-01 15:33:35,805] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8647b7e037124214af5d243cca5761b6job_title'
[2022-07-01 15:33:35,807] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8647b7e037124214af5d243cca5761b6job_title%27
[2022-07-01 15:33:35,814] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.694875ms
[2022-07-01 15:33:36,314] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 15:33:36,339] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:33:37,188] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:33:37,232] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-07-01 15:33:37,233] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:33:37,258] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-07-01 15:33:37,258] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-07-01 15:33:37,259] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2460398988, 0, b'')
[2022-07-01 15:33:37,817] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:37,820] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:33:37,822] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 15:33:37,826] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.207362ms
[2022-07-01 15:33:37,826] - [task:3216] INFO - {'requestID': '39bc2de0-7f20-4cfc-b97e-0e65f0f1f84b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8647b7e037124214af5d243cca5761b6job_title', 'index_id': '1e4bddac31af1f18', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.207362ms', 'executionTime': '2.127169ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 15:33:37,827] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:37,827] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:37,827] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:33:37,828] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:33:37,828] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 15:33:37,828] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:33:37,829] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:33:37,829] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:33:37,829] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:33:38,828] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:38,833] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:33:38,836] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 15:33:38,903] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.006531ms
[2022-07-01 15:33:38,904] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:33:38,904] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:33:39,691] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:39,691] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:33:40,697] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8647b7e037124214af5d243cca5761b6job_title'
[2022-07-01 15:33:40,700] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8647b7e037124214af5d243cca5761b6job_title%27
[2022-07-01 15:33:40,708] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.362111ms
[2022-07-01 15:33:40,710] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8647b7e037124214af5d243cca5761b6job_title ON default USING GSI
[2022-07-01 15:33:40,712] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8647b7e037124214af5d243cca5761b6job_title+ON+default+USING+GSI
[2022-07-01 15:33:40,750] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.067764ms
[2022-07-01 15:33:40,756] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8647b7e037124214af5d243cca5761b6job_title'
[2022-07-01 15:33:40,759] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8647b7e037124214af5d243cca5761b6job_title%27
[2022-07-01 15:33:40,763] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.477377ms
[2022-07-01 15:33:40,763] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3fadc689-d554-4d18-a1ff-e22b56d239b2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.477377ms', 'executionTime': '1.405644ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:33:40,772] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:40,776] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:40,776] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:41,282] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:41,290] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:33:41,290] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:33:41,362] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:33:41,371] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:41,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:41,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:41,849] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:41,860] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:33:41,860] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:33:41,945] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:33:41,953] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:33:41,953] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:33:41,953] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:33:41,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:33:41,958] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:33:41,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.024278ms
[2022-07-01 15:33:41,969] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:33:41,971] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:33:42,025] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.2668ms
[2022-07-01 15:33:42,041] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:33:42,042] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.63315871747074, 'mem_free': 13110013952, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:33:42,042] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:33:42,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:42,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:42,561] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:42,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:42,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:43,028] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:43,033] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:43,034] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:43,688] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:43,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:43,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:44,482] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:49,628] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:33:49,653] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:33:50,654] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:33:50,657] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:33:50,658] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:33:50,664] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:33:50,670] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:33:50,676] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:33:50,678] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:33:50,687] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:33:50,688] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:33:50,690] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:50,702] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:33:50,702] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:50,706] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:33:50,708] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:33:50,708] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:50,711] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:33:50,713] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:33:50,714] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:50,716] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:33:50,718] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:33:50,718] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:50,722] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:33:50,722] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:33:50,722] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:33:50,723] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 75.515s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_5'}
[2022-07-01 15:33:50,791] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:50,792] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:51,225] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:51,234] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:33:51,242] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:33:51,242] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-07-01 15:33:51,243] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:33:51,793] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:33:51,797] - [task:161] INFO -  {'uptime': '377', 'memoryTotal': 15466930176, 'memoryFree': 13273042944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:33:51,799] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:33:51,799] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:33:51,800] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:33:51,805] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:33:51,813] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:33:51,814] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:33:51,820] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:33:51,820] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:33:51,821] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:33:51,821] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:33:51,824] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:33:51,827] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:51,827] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:52,260] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:52,261] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:33:52,326] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:33:52,327] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:33:52,332] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:33:52,333] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:33:52,360] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:33:52,364] - [task:161] INFO -  {'uptime': '373', 'memoryTotal': 15466930176, 'memoryFree': 13191643136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:33:52,366] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:33:52,371] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:33:52,371] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:33:52,373] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:33:52,377] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:52,377] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:52,810] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:52,811] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:33:52,876] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:33:52,877] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:33:52,882] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:33:52,883] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:33:52,908] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:33:52,912] - [task:161] INFO -  {'uptime': '374', 'memoryTotal': 15466930176, 'memoryFree': 13189586944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:33:52,914] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:33:52,918] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:33:52,918] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:33:52,921] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:33:52,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:52,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:53,375] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:53,376] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:33:53,445] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:33:53,446] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:33:53,451] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:33:53,452] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:33:53,473] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:33:53,475] - [task:161] INFO -  {'uptime': '374', 'memoryTotal': 15466930176, 'memoryFree': 13274640384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:33:53,478] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:33:53,483] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:33:53,484] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:33:53,486] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:33:53,490] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:53,490] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:53,916] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:53,917] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:33:53,987] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:33:53,988] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:33:53,993] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:33:53,994] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:33:54,013] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:33:54,145] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:33:59,147] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:33:59,158] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:33:59,164] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:33:59,165] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:33:59,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:33:59,606] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:33:59,672] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:33:59,673] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:33:59,673] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:34:00,034] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:34:00,065] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:34:00,066] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:34:00,645] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:34:00,809] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:34:00,861] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:34:00,863] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:34:00,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:34:00,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:34:01,314] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:34:01,319] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:34:01,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:34:01,734] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:34:01,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:34:01,743] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:34:02,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:34:02,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:34:02,440] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:34:03,251] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:34:08,445] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:34:08,445] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.23001206518701, 'mem_free': 13273890816, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:34:08,445] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:34:08,446] - [basetestcase:467] INFO - Time to execute basesetup : 17.656761646270752
[2022-07-01 15:34:08,450] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:34:08,450] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:34:08,456] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:34:08,456] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:34:08,463] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:34:08,463] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:34:08,470] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:34:08,470] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:34:08,474] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:34:08,486] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:34:08,486] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:34:08,486] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:34:13,498] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:34:13,501] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:34:13,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:34:13,916] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:34:14,878] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:34:15,009] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:34:16,838] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:34:16,854] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-01 15:34:17,896] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:34:17,896] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:34:17,896] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:34:47,926] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:34:47,932] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:34:47,934] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:34:47,956] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.19986ms
[2022-07-01 15:34:47,956] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '80c0d215-e5bf-4a45-bda2-d7b1299751f9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.19986ms', 'executionTime': '20.124368ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:34:47,957] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:34:47,959] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:34:47,961] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:34:48,353] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 389.486598ms
[2022-07-01 15:34:48,353] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:34:48,358] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:34:48,361] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:34:48,370] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.28101ms
[2022-07-01 15:34:48,398] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:34:48,402] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:34:48,426] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:34:48,426] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:34:48,441] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:34:48,454] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:34:49,395] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeef5f407d52e14f82a696b9b06b1cbc45job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:34:49,398] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeef5f407d52e14f82a696b9b06b1cbc45job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:34:49,426] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.407815ms
[2022-07-01 15:34:49,427] - [base_gsi:282] INFO - BUILD INDEX on default(employeeef5f407d52e14f82a696b9b06b1cbc45job_title) USING GSI
[2022-07-01 15:34:50,433] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeef5f407d52e14f82a696b9b06b1cbc45job_title) USING GSI
[2022-07-01 15:34:50,435] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeef5f407d52e14f82a696b9b06b1cbc45job_title%29+USING+GSI
[2022-07-01 15:34:50,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.855184ms
[2022-07-01 15:34:51,466] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeef5f407d52e14f82a696b9b06b1cbc45job_title'
[2022-07-01 15:34:51,468] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeef5f407d52e14f82a696b9b06b1cbc45job_title%27
[2022-07-01 15:34:51,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.260957ms
[2022-07-01 15:34:52,481] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeef5f407d52e14f82a696b9b06b1cbc45job_title'
[2022-07-01 15:34:52,484] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeef5f407d52e14f82a696b9b06b1cbc45job_title%27
[2022-07-01 15:34:52,490] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.307683ms
[2022-07-01 15:34:53,015] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:34:53,039] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:34:55,643] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:34:56,494] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:34:56,502] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:34:56,504] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 15:34:56,509] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.631464ms
[2022-07-01 15:34:56,509] - [task:3216] INFO - {'requestID': 'aef59bab-21cb-47a5-b110-00c31f0b9d96', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeef5f407d52e14f82a696b9b06b1cbc45job_title', 'index_id': '34e599644308f6b9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.631464ms', 'executionTime': '3.551786ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 15:34:56,510] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:34:56,510] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:34:56,510] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:34:56,511] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:34:56,511] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 15:34:56,511] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:34:56,512] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:34:56,512] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:34:56,512] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:34:57,512] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:34:57,517] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:34:57,519] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 15:34:57,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 104.591529ms
[2022-07-01 15:34:57,638] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:34:57,639] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:34:59,179] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:34:59,179] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:35:00,187] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeef5f407d52e14f82a696b9b06b1cbc45job_title'
[2022-07-01 15:35:00,189] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeef5f407d52e14f82a696b9b06b1cbc45job_title%27
[2022-07-01 15:35:00,199] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.287324ms
[2022-07-01 15:35:00,201] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeef5f407d52e14f82a696b9b06b1cbc45job_title ON default USING GSI
[2022-07-01 15:35:00,203] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeef5f407d52e14f82a696b9b06b1cbc45job_title+ON+default+USING+GSI
[2022-07-01 15:35:00,246] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.70725ms
[2022-07-01 15:35:00,252] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeef5f407d52e14f82a696b9b06b1cbc45job_title'
[2022-07-01 15:35:00,255] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeef5f407d52e14f82a696b9b06b1cbc45job_title%27
[2022-07-01 15:35:00,259] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.616301ms
[2022-07-01 15:35:00,259] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '42723708-c67f-4460-af67-780641e5cae4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.616301ms', 'executionTime': '1.539152ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:35:00,268] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:35:00,272] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:00,272] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:00,861] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:00,869] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:35:00,869] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:35:00,958] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:35:00,966] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:35:00,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:00,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:01,510] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:01,521] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:35:01,521] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:35:01,608] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:35:01,616] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:35:01,617] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:35:01,617] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:35:01,619] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:35:01,622] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:35:01,629] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.640857ms
[2022-07-01 15:35:01,631] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:35:01,634] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:35:01,697] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.415995ms
[2022-07-01 15:35:01,709] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:35:01,710] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.95153249814432, 'mem_free': 13105799168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:35:01,710] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:35:01,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:01,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:02,279] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:02,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:02,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:03,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:03,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:03,065] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:03,984] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:03,995] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:03,995] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:04,921] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:10,574] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:35:10,597] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:35:11,459] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:35:11,462] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:35:11,462] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:35:11,468] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:35:11,474] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:35:11,480] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:35:11,481] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:35:11,491] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:35:11,492] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:35:11,494] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:11,506] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:35:11,506] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:11,509] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:35:11,512] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:35:11,512] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:11,516] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:35:11,518] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:35:11,518] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:11,522] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:35:11,524] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:35:11,524] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:11,527] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:35:11,528] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:35:11,528] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:35:11,528] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 80.745s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_6'}
[2022-07-01 15:35:11,588] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:11,588] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:12,126] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:12,132] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:35:12,141] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:35:12,141] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-07-01 15:35:12,142] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:35:12,592] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:35:12,596] - [task:161] INFO -  {'uptime': '458', 'memoryTotal': 15466930176, 'memoryFree': 13208596480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:35:12,598] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:35:12,598] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:35:12,598] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:35:12,606] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:35:12,614] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:35:12,615] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:35:12,618] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:35:12,619] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:35:12,619] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:35:12,619] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:35:12,622] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:35:12,626] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:12,626] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:13,139] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:13,140] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:35:13,219] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:35:13,220] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:35:13,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:35:13,226] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:35:13,260] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:35:13,264] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13205655552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:35:13,266] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:35:13,270] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:35:13,271] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:35:13,273] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:35:13,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:13,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:13,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:13,785] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:35:13,860] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:35:13,861] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:35:13,866] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:35:13,867] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:35:13,888] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:35:13,892] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13206167552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:35:13,894] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:35:13,899] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:35:13,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:35:13,903] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:35:13,909] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:13,909] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:14,408] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:14,409] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:35:14,485] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:35:14,486] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:35:14,491] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:35:14,491] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:35:14,513] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:35:14,516] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13209812992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:35:14,518] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:35:14,522] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:35:14,522] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:35:14,525] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:35:14,528] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:14,529] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:15,044] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:15,045] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:35:15,126] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:35:15,127] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:35:15,132] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:35:15,133] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:35:15,149] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:35:15,256] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:35:20,261] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:35:20,274] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:35:20,280] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:20,280] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:20,827] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:20,828] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:35:20,915] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:35:20,916] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:35:20,917] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:35:21,167] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:35:21,198] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:35:21,198] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:35:21,767] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:35:21,807] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:35:21,852] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:35:21,853] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:35:21,860] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:21,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:22,428] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:22,433] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:22,433] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:23,042] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:23,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:23,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:23,927] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:23,939] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:23,940] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:24,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:30,587] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:35:30,587] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.08350533796084, 'mem_free': 13216530432, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:35:30,587] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:35:30,588] - [basetestcase:467] INFO - Time to execute basesetup : 19.001858711242676
[2022-07-01 15:35:30,592] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:35:30,592] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:35:30,598] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:35:30,599] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:35:30,604] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:35:30,605] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:35:30,611] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:35:30,611] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:35:30,615] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:35:30,630] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:35:30,631] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:35:30,631] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:35:35,643] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:35:35,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:35:35,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:35:36,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:35:37,193] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:35:37,342] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:35:39,103] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:35:39,121] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-01 15:35:40,491] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:35:40,492] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:35:40,492] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:36:10,503] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:36:10,509] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:36:10,511] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:36:10,532] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.349994ms
[2022-07-01 15:36:10,533] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e792b4af-f934-412b-b547-a704d1bacd43', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.349994ms', 'executionTime': '19.211763ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:36:10,533] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:36:10,535] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:36:10,537] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:36:10,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 357.418633ms
[2022-07-01 15:36:10,896] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:36:10,903] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:36:10,916] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:36:10,925] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.942979ms
[2022-07-01 15:36:10,973] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:36:10,978] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:36:10,996] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:36:10,997] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:36:11,009] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:36:11,019] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:36:11,971] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee56bc88011e5426c826b2babaaaf4bb1job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:36:11,974] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee56bc88011e5426c826b2babaaaf4bb1job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:36:12,012] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.780466ms
[2022-07-01 15:36:12,012] - [base_gsi:282] INFO - BUILD INDEX on default(employeee56bc88011e5426c826b2babaaaf4bb1job_title) USING GSI
[2022-07-01 15:36:13,018] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee56bc88011e5426c826b2babaaaf4bb1job_title) USING GSI
[2022-07-01 15:36:13,021] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee56bc88011e5426c826b2babaaaf4bb1job_title%29+USING+GSI
[2022-07-01 15:36:13,053] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.625957ms
[2022-07-01 15:36:14,060] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee56bc88011e5426c826b2babaaaf4bb1job_title'
[2022-07-01 15:36:14,062] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee56bc88011e5426c826b2babaaaf4bb1job_title%27
[2022-07-01 15:36:14,071] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.212063ms
[2022-07-01 15:36:15,077] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee56bc88011e5426c826b2babaaaf4bb1job_title'
[2022-07-01 15:36:15,079] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee56bc88011e5426c826b2babaaaf4bb1job_title%27
[2022-07-01 15:36:15,088] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.572731ms
[2022-07-01 15:36:15,625] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 15:36:15,649] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:36:16,825] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:36:17,500] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-01 15:36:17,527] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:36:18,880] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:36:19,276] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:36:19,306] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:36:21,991] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:36:22,105] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:22,112] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:36:22,114] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 15:36:22,119] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.085815ms
[2022-07-01 15:36:22,119] - [task:3216] INFO - {'requestID': '61cbffdc-0b4c-4623-a8db-00ddb2666e51', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee56bc88011e5426c826b2babaaaf4bb1job_title', 'index_id': 'aed5ff523a81d10b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.085815ms', 'executionTime': '3.01448ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 15:36:22,119] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:22,120] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:22,120] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:36:22,120] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:36:22,121] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 15:36:22,121] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:36:22,121] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:36:22,122] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:36:22,122] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 15:36:23,121] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:23,127] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 15:36:23,130] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 15:36:23,494] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 101.923362ms
[2022-07-01 15:36:23,494] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:36:23,495] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:36:25,063] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:25,063] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 15:36:26,071] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee56bc88011e5426c826b2babaaaf4bb1job_title'
[2022-07-01 15:36:26,073] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee56bc88011e5426c826b2babaaaf4bb1job_title%27
[2022-07-01 15:36:26,083] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.454945ms
[2022-07-01 15:36:26,085] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee56bc88011e5426c826b2babaaaf4bb1job_title ON default USING GSI
[2022-07-01 15:36:26,087] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee56bc88011e5426c826b2babaaaf4bb1job_title+ON+default+USING+GSI
[2022-07-01 15:36:26,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.527803ms
[2022-07-01 15:36:26,142] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee56bc88011e5426c826b2babaaaf4bb1job_title'
[2022-07-01 15:36:26,144] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee56bc88011e5426c826b2babaaaf4bb1job_title%27
[2022-07-01 15:36:26,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.353252ms
[2022-07-01 15:36:26,148] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd5298b74-bc2a-4b46-8d2f-7d6e0150bf45', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.353252ms', 'executionTime': '1.286835ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:36:26,156] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:36:26,159] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:26,159] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:26,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:26,846] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:36:26,846] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:36:26,947] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:36:26,954] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:36:26,957] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:26,957] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:27,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:27,661] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:36:27,662] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:36:27,765] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:36:27,773] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:36:27,773] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:36:27,773] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:36:27,776] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:36:27,778] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:36:27,787] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.007114ms
[2022-07-01 15:36:27,789] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:36:27,791] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:36:27,847] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.207814ms
[2022-07-01 15:36:27,858] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:36:27,859] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.60835808878605, 'mem_free': 12998266880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:36:27,859] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:36:27,863] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:27,863] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:28,521] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:28,526] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:28,526] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:29,345] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:29,353] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:29,353] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:30,486] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:30,493] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:30,494] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:31,644] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:39,000] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:36:39,023] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:36:39,657] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:36:39,661] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:36:39,661] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:36:39,667] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:36:39,674] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:36:39,681] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:36:39,682] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:36:39,691] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:36:39,692] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:36:39,694] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:39,706] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:36:39,706] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:39,710] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:36:39,713] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:36:39,713] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:39,716] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:36:39,718] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:36:39,718] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:39,721] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:36:39,723] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:36:39,724] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:39,727] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:36:39,727] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 15:36:39,727] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:36:39,728] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 88.147s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_7'}
[2022-07-01 15:36:39,817] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:39,817] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:40,488] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:40,493] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:36:40,502] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:36:40,502] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-07-01 15:36:40,503] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:36:40,820] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:36:40,823] - [task:161] INFO -  {'uptime': '546', 'memoryTotal': 15466930176, 'memoryFree': 13157474304, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:36:40,825] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:36:40,825] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:36:40,826] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:36:40,832] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:36:40,845] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:36:40,846] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:36:40,850] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:36:40,852] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:36:40,853] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:36:40,853] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:36:40,855] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:36:40,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:40,859] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:41,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:41,484] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:36:41,582] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:36:41,583] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:36:41,588] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:36:41,588] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:36:41,613] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:36:41,616] - [task:161] INFO -  {'uptime': '544', 'memoryTotal': 15466930176, 'memoryFree': 13156933632, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:36:41,619] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:36:41,623] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:36:41,624] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:36:41,626] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:36:41,630] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:41,630] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:42,242] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:42,243] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:36:42,347] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:36:42,348] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:36:42,353] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:36:42,354] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:36:42,376] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:36:42,379] - [task:161] INFO -  {'uptime': '545', 'memoryTotal': 15466930176, 'memoryFree': 13156593664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:36:42,381] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:36:42,385] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:36:42,386] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:36:42,389] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:36:42,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:42,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:42,989] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:42,990] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:36:43,085] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:36:43,086] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:36:43,091] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:36:43,091] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:36:43,115] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:36:43,118] - [task:161] INFO -  {'uptime': '539', 'memoryTotal': 15466930176, 'memoryFree': 13156511744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:36:43,120] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:36:43,124] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:36:43,124] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:36:43,126] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:36:43,130] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:43,131] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:43,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:43,725] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:36:43,816] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:36:43,817] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:36:43,822] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:36:43,823] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:36:43,844] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:36:43,948] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:36:48,954] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:36:48,965] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:36:48,971] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:48,972] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:49,619] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:49,620] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:36:49,727] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:36:49,728] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:36:49,728] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:36:49,859] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:36:49,889] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:36:49,890] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:36:50,452] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:36:50,496] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:36:50,540] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:36:50,542] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-01 15:36:50,549] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:50,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:51,192] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:51,197] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:51,197] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:52,073] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:52,084] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:52,085] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:53,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:36:53,169] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:36:53,169] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:36:54,245] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:00,728] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:37:00,729] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 29.8418402250418, 'mem_free': 13171474432, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:37:00,729] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:37:00,729] - [basetestcase:467] INFO - Time to execute basesetup : 20.91506862640381
[2022-07-01 15:37:00,734] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:37:00,734] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:37:00,740] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:37:00,741] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:37:00,747] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:37:00,748] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:37:00,754] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:37:00,754] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:37:00,764] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:00,775] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:37:00,776] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:37:00,776] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:37:05,787] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:37:05,790] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:05,791] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:06,419] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:07,447] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:37:07,473] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:37:10,117] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:37:10,148] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:10,148] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:37:10,149] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:37:40,169] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:37:40,174] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:37:40,176] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:37:40,197] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.489929ms
[2022-07-01 15:37:40,198] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '762fc8a7-4793-4152-9aa8-1f675864079a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.489929ms', 'executionTime': '19.405895ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:37:40,198] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:37:40,200] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:37:40,202] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:37:40,580] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 375.952282ms
[2022-07-01 15:37:40,580] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:37:40,594] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:37:40,599] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:37:40,607] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.555819ms
[2022-07-01 15:37:40,635] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:40,640] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:37:40,661] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:37:40,663] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:37:40,679] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:37:40,680] - [base_gsi:326] INFO - []
[2022-07-01 15:37:41,634] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:37:41,636] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:37:41,679] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.905385ms
[2022-07-01 15:37:41,687] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:37:41,690] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:37:41,723] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.78095ms
[2022-07-01 15:37:41,724] - [base_gsi:282] INFO - BUILD INDEX on default(employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title,employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr) USING GSI
[2022-07-01 15:37:42,730] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title,employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr) USING GSI
[2022-07-01 15:37:42,732] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%2Cemployeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr%29+USING+GSI
[2022-07-01 15:37:42,769] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.497463ms
[2022-07-01 15:37:43,775] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title'
[2022-07-01 15:37:43,778] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%27
[2022-07-01 15:37:43,787] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.328ms
[2022-07-01 15:37:44,793] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title'
[2022-07-01 15:37:44,795] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%27
[2022-07-01 15:37:44,804] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.352231ms
[2022-07-01 15:37:44,806] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr'
[2022-07-01 15:37:44,808] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr%27
[2022-07-01 15:37:44,811] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.261156ms
[2022-07-01 15:37:45,817] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title'
[2022-07-01 15:37:45,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%27
[2022-07-01 15:37:45,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.528811ms
[2022-07-01 15:37:45,831] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title ON default USING GSI
[2022-07-01 15:37:45,833] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title+ON+default+USING+GSI
[2022-07-01 15:37:45,878] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.241114ms
[2022-07-01 15:37:45,885] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr'
[2022-07-01 15:37:45,888] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr%27
[2022-07-01 15:37:45,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.767758ms
[2022-07-01 15:37:45,895] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr ON default USING GSI
[2022-07-01 15:37:45,897] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr+ON+default+USING+GSI
[2022-07-01 15:37:45,936] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.54481ms
[2022-07-01 15:37:45,943] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title'
[2022-07-01 15:37:45,946] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02job_title%27
[2022-07-01 15:37:45,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.792606ms
[2022-07-01 15:37:45,955] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '54d24367-06d3-4b76-8618-b1c0278ba97e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.792606ms', 'executionTime': '7.703596ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:37:45,960] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr'
[2022-07-01 15:37:45,963] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee4bc20dbc21c42c2b9d9adda01f2bf02join_yr%27
[2022-07-01 15:37:45,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.237339ms
[2022-07-01 15:37:45,967] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a996d79a-134b-42fb-8835-2ee170c7c36a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.237339ms', 'executionTime': '1.19257ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:37:45,974] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:45,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:45,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:46,626] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:46,634] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:37:46,635] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:37:46,733] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:37:46,740] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:46,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:46,743] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:47,367] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:47,378] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:37:47,378] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:37:47,478] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:37:47,486] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:37:47,486] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:37:47,486] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:37:47,488] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:37:47,491] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:37:47,501] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.40399ms
[2022-07-01 15:37:47,503] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:37:47,505] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:37:47,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.577573ms
[2022-07-01 15:37:47,574] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:37:47,574] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.633149946974125, 'mem_free': 13065269248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:37:47,575] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:37:47,582] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:47,582] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:48,216] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:48,222] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:48,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:48,903] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:48,908] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:48,908] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:50,044] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:50,052] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:50,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:37:51,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:37:58,196] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-07-01 15:37:58,222] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:37:59,344] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:37:59,346] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:37:59,347] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:37:59,353] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:37:59,359] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:37:59,365] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:37:59,366] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:37:59,374] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:37:59,375] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:37:59,377] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:37:59,390] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:37:59,390] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:37:59,394] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:37:59,396] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:37:59,396] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:37:59,402] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:37:59,407] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:37:59,407] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:37:59,411] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:37:59,413] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:37:59,413] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:37:59,417] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:37:59,417] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-01 15:37:59,417] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:37:59,418] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 79.609s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_8'}
[2022-07-01 15:37:59,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:37:59,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:00,138] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:00,146] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:38:00,155] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:38:00,155] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-07-01 15:38:00,156] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:38:00,452] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:38:00,456] - [task:161] INFO -  {'uptime': '626', 'memoryTotal': 15466930176, 'memoryFree': 13179146240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:38:00,458] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:38:00,458] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:38:00,459] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:38:00,468] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:38:00,476] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:38:00,476] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:38:00,478] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:38:00,479] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:38:00,479] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:38:00,480] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:38:00,482] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:38:00,486] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:00,486] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:01,131] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:01,132] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:38:01,222] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:38:01,223] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:38:01,228] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:38:01,229] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:38:01,255] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:38:01,258] - [task:161] INFO -  {'uptime': '625', 'memoryTotal': 15466930176, 'memoryFree': 13177376768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:38:01,261] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:38:01,266] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:38:01,266] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:38:01,272] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:38:01,275] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:01,275] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:01,915] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:01,916] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:38:02,016] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:38:02,017] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:38:02,022] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:38:02,023] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:38:02,048] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:38:02,051] - [task:161] INFO -  {'uptime': '620', 'memoryTotal': 15466930176, 'memoryFree': 13177683968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:38:02,053] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:38:02,059] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:38:02,060] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:38:02,062] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:38:02,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:02,070] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:02,712] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:02,713] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:38:02,819] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:38:02,820] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:38:02,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:38:02,826] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:38:02,847] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:38:02,850] - [task:161] INFO -  {'uptime': '620', 'memoryTotal': 15466930176, 'memoryFree': 13177278464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:38:02,852] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:38:02,856] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:38:02,857] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:38:02,861] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:38:02,865] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:02,865] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:03,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:03,486] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:38:03,577] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:38:03,578] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:38:03,583] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:38:03,584] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:38:03,599] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:38:03,703] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:38:08,707] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:38:08,720] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:38:08,725] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:08,725] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:09,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:09,347] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:38:09,444] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:38:09,445] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:38:09,446] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:38:09,615] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:38:09,647] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:38:09,648] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:38:10,211] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:38:10,432] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:38:10,911] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:38:10,914] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-01 15:38:10,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:10,921] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:11,570] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:11,575] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:11,575] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:12,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:12,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:12,379] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:13,447] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:13,459] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:13,461] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:14,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:21,390] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:38:21,391] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 29.92047307993748, 'mem_free': 13174394880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:38:21,391] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:38:21,391] - [basetestcase:467] INFO - Time to execute basesetup : 21.943670511245728
[2022-07-01 15:38:21,396] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:38:21,396] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:38:21,402] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:38:21,402] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:38:21,408] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:38:21,408] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:38:21,415] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:38:21,415] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:38:21,419] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:38:21,430] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:38:21,430] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:38:21,430] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:38:26,438] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:38:26,441] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:38:26,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:38:27,089] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:38:28,124] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:38:28,150] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:38:30,762] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:38:30,797] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:38:30,798] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:38:30,798] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:39:00,828] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:39:00,834] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:39:00,836] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:39:00,858] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.144435ms
[2022-07-01 15:39:00,858] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c0fe763f-d634-4ff6-8f4a-112fd19534ed', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.144435ms', 'executionTime': '20.075641ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:39:00,859] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:39:00,861] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:39:00,863] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:39:01,217] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 351.371858ms
[2022-07-01 15:39:01,217] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:39:01,227] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:39:01,231] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:39:01,242] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.257516ms
[2022-07-01 15:39:01,268] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:01,272] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:39:01,290] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:39:01,291] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:39:01,305] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:39:01,306] - [base_gsi:326] INFO - []
[2022-07-01 15:39:02,265] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:39:02,267] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:39:02,297] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.858606ms
[2022-07-01 15:39:02,298] - [base_gsi:282] INFO - BUILD INDEX on default(employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr) USING GSI
[2022-07-01 15:39:03,304] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr) USING GSI
[2022-07-01 15:39:03,306] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%29+USING+GSI
[2022-07-01 15:39:03,330] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.529526ms
[2022-07-01 15:39:04,337] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr'
[2022-07-01 15:39:04,339] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%27
[2022-07-01 15:39:04,349] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.019681ms
[2022-07-01 15:39:05,355] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr'
[2022-07-01 15:39:05,357] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%27
[2022-07-01 15:39:05,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.280539ms
[2022-07-01 15:39:06,374] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr'
[2022-07-01 15:39:06,376] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%27
[2022-07-01 15:39:06,383] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.868987ms
[2022-07-01 15:39:06,386] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr ON default USING GSI
[2022-07-01 15:39:06,389] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr+ON+default+USING+GSI
[2022-07-01 15:39:06,449] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.121974ms
[2022-07-01 15:39:06,456] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr'
[2022-07-01 15:39:06,458] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee5e47be2e7794fcea4489e6fb56dfd1ejob_title_join_yr%27
[2022-07-01 15:39:06,461] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.544967ms
[2022-07-01 15:39:06,462] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0cb698c5-63db-4546-800e-366eebd404b8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.544967ms', 'executionTime': '1.485059ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:39:06,470] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:06,473] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:06,473] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:07,169] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:07,177] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:39:07,177] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:39:07,291] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:39:07,299] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:07,301] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:07,302] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:07,924] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:07,937] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:39:07,937] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:39:08,042] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:39:08,051] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:08,051] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:39:08,051] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:39:08,053] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:39:08,056] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:39:08,066] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.513535ms
[2022-07-01 15:39:08,068] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:39:08,070] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:39:08,113] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.186286ms
[2022-07-01 15:39:08,127] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:39:08,127] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.316565829086527, 'mem_free': 13050499072, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:39:08,127] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:39:08,134] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:08,134] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:08,757] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:08,762] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:08,763] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:09,385] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:09,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:09,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:10,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:10,401] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:10,402] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:11,516] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:18,673] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-07-01 15:39:18,700] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:39:19,043] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:39:19,046] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:39:19,046] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:39:19,053] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:39:19,059] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:39:19,065] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:39:19,066] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:39:19,075] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:39:19,076] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:39:19,078] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,090] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:39:19,090] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,094] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:39:19,097] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:39:19,097] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,101] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:39:19,103] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:39:19,103] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,107] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:39:19,109] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:39:19,109] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,112] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:39:19,113] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-01 15:39:19,113] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:39:19,113] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 79.671s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_9'}
[2022-07-01 15:39:19,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:19,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:19,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:19,803] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:39:19,813] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:39:19,813] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-07-01 15:39:19,814] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:39:20,156] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:39:20,159] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13162508288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:39:20,161] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:39:20,162] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:39:20,162] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:39:20,171] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:39:20,185] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:39:20,186] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:39:20,188] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:39:20,189] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:39:20,189] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:39:20,190] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:39:20,192] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:39:20,196] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:20,196] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:20,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:20,854] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:39:20,956] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:39:20,957] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:39:20,962] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:39:20,963] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:39:20,996] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:39:20,999] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13163438080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:39:21,002] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:39:21,008] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:39:21,008] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:39:21,010] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:39:21,014] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:21,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:21,661] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:21,663] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:39:21,763] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:39:21,764] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:39:21,769] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:39:21,770] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:39:21,788] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:39:21,791] - [task:161] INFO -  {'uptime': '700', 'memoryTotal': 15466930176, 'memoryFree': 13162491904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:39:21,793] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:39:21,798] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:39:21,799] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:39:21,801] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:39:21,807] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:21,808] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:22,475] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:22,477] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:39:22,578] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:39:22,579] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:39:22,584] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:39:22,585] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:39:22,606] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:39:22,609] - [task:161] INFO -  {'uptime': '700', 'memoryTotal': 15466930176, 'memoryFree': 13163044864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:39:22,611] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:39:22,615] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:39:22,615] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:39:22,618] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:39:22,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:22,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:23,316] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:23,317] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:39:23,417] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:39:23,418] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:39:23,423] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:39:23,424] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:39:23,439] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:39:23,555] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:39:28,560] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:39:28,572] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:39:28,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:28,577] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:29,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:29,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:39:29,352] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:39:29,353] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:39:29,354] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:39:29,456] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:39:29,486] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:39:29,487] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:39:30,233] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:39:30,279] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:39:30,338] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:39:30,341] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-01 15:39:30,347] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:30,347] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:31,026] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:31,032] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:31,032] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:31,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:31,967] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:31,967] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:33,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:33,114] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:33,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:34,230] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:40,883] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:39:40,884] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.48107842642824, 'mem_free': 13162823680, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:39:40,884] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:39:40,884] - [basetestcase:467] INFO - Time to execute basesetup : 21.735870122909546
[2022-07-01 15:39:40,890] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:39:40,890] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:39:40,897] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:39:40,897] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:39:40,905] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:39:40,905] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:39:40,911] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:39:40,912] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:39:40,916] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:40,928] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:39:40,928] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:39:40,928] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:39:45,941] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:39:45,944] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:39:45,944] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:39:46,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:39:47,740] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:39:47,766] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:39:50,288] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:39:50,321] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:39:50,321] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:39:50,322] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:40:20,350] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:40:20,356] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:40:20,358] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:40:20,379] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.680189ms
[2022-07-01 15:40:20,380] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '61435044-946d-4885-b167-5801450ebec5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.680189ms', 'executionTime': '19.605541ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:40:20,380] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:40:20,382] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:40:20,384] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:40:20,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 373.090663ms
[2022-07-01 15:40:20,759] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:40:20,772] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:40:20,784] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:40:20,797] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.330227ms
[2022-07-01 15:40:20,821] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:20,824] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:40:20,842] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:40:20,843] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:40:20,874] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:40:20,890] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:21,821] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3b9904c9718142508ec32fafd7198d6ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:40:21,824] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3b9904c9718142508ec32fafd7198d6ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:40:21,865] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.203978ms
[2022-07-01 15:40:21,865] - [base_gsi:282] INFO - BUILD INDEX on default(`employee3b9904c9718142508ec32fafd7198d6ajoin_yr`) USING GSI
[2022-07-01 15:40:22,871] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee3b9904c9718142508ec32fafd7198d6ajoin_yr`) USING GSI
[2022-07-01 15:40:22,874] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee3b9904c9718142508ec32fafd7198d6ajoin_yr%60%29+USING+GSI
[2022-07-01 15:40:22,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.179165ms
[2022-07-01 15:40:22,913] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b9904c9718142508ec32fafd7198d6ajoin_yr'
[2022-07-01 15:40:22,916] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b9904c9718142508ec32fafd7198d6ajoin_yr%27
[2022-07-01 15:40:22,928] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.815571ms
[2022-07-01 15:40:23,934] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b9904c9718142508ec32fafd7198d6ajoin_yr'
[2022-07-01 15:40:23,936] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b9904c9718142508ec32fafd7198d6ajoin_yr%27
[2022-07-01 15:40:23,948] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.452575ms
[2022-07-01 15:40:24,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b9904c9718142508ec32fafd7198d6ajoin_yr'
[2022-07-01 15:40:24,958] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b9904c9718142508ec32fafd7198d6ajoin_yr%27
[2022-07-01 15:40:24,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.070819ms
[2022-07-01 15:40:25,972] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:40:25,974] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 15:40:25,980] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.835515ms
[2022-07-01 15:40:25,980] - [base_gsi:504] INFO - {'requestID': 'b1ae7556-a3a1-4889-9d9a-2ec572d9a2dd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3b9904c9718142508ec32fafd7198d6ajoin_yr', 'index_id': 'c7babfc8e0a294cf', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.835515ms', 'executionTime': '3.750896ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 15:40:25,981] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:40:25,981] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:40:25,981] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:40:25,981] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:40:25,982] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:40:25,983] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:40:25,983] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:40:25,984] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:40:26,051] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:40:26,065] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:40:26,067] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 15:40:26,175] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 93.806605ms
[2022-07-01 15:40:26,175] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:40:26,176] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:40:27,890] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:27,892] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3b9904c9718142508ec32fafd7198d6ajoin_yr'
[2022-07-01 15:40:27,895] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3b9904c9718142508ec32fafd7198d6ajoin_yr%27
[2022-07-01 15:40:27,900] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.044287ms
[2022-07-01 15:40:27,900] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e36a4b29-3999-4ac6-ac24-1cb6ceddddd3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.044287ms', 'executionTime': '3.95877ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:40:27,907] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:27,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:27,910] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:28,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:28,639] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:40:28,639] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:40:28,777] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:40:28,784] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:28,787] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:28,787] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:29,444] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:29,455] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:40:29,455] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:40:29,564] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:40:29,573] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:40:29,573] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:40:29,573] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:40:29,575] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:40:29,578] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:40:29,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.537598ms
[2022-07-01 15:40:29,586] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7f9f332d-b8df-4500-8ff9-48b640098026', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.537598ms', 'executionTime': '6.46344ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:40:29,590] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:40:29,590] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.79582163749941, 'mem_free': 13005946880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:40:29,590] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:40:29,594] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:29,594] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:30,262] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:30,267] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:30,267] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:30,927] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:30,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:30,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:31,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:31,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:31,978] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:33,141] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:40,580] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-07-01 15:40:40,594] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:40:40,601] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:40:40,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:40:40,613] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:40:40,613] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:40:40,619] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:40:40,621] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:40:40,623] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:40,635] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:40:40,635] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:40,638] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:40:40,640] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:40:40,640] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:40,643] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:40:40,645] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:40:40,646] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:40,648] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:40:40,650] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:40:40,650] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:40,654] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:40:40,654] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-01 15:40:40,654] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:40:40,655] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 81.512s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_10'}
[2022-07-01 15:40:40,690] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:40,690] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:41,379] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:41,385] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:40:41,394] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:40:41,394] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-07-01 15:40:41,395] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:40:41,695] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:40:41,699] - [task:161] INFO -  {'uptime': '784', 'memoryTotal': 15466930176, 'memoryFree': 13166362624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:40:41,701] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:40:41,701] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:40:41,702] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:40:41,706] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:40:41,720] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:40:41,720] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:40:41,723] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:40:41,724] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:40:41,724] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:40:41,724] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:40:41,728] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:40:41,731] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:41,732] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:42,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:42,417] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:40:42,514] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:40:42,515] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:40:42,521] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:40:42,521] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:40:42,547] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:40:42,550] - [task:161] INFO -  {'uptime': '785', 'memoryTotal': 15466930176, 'memoryFree': 13165125632, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:40:42,552] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:40:42,557] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:40:42,557] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:40:42,560] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:40:42,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:42,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:43,254] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:43,255] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:40:43,362] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:40:43,363] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:40:43,368] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:40:43,369] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:40:43,391] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:40:43,394] - [task:161] INFO -  {'uptime': '786', 'memoryTotal': 15466930176, 'memoryFree': 13164511232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:40:43,397] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:40:43,401] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:40:43,401] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:40:43,404] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:40:43,408] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:43,408] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:44,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:44,078] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:40:44,177] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:40:44,178] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:40:44,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:40:44,184] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:40:44,209] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:40:44,212] - [task:161] INFO -  {'uptime': '780', 'memoryTotal': 15466930176, 'memoryFree': 13165051904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:40:44,215] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:40:44,220] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:40:44,220] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:40:44,223] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:40:44,227] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:44,228] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:44,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:44,926] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:40:45,030] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:40:45,031] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:40:45,036] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:40:45,037] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:40:45,052] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:40:45,176] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:40:50,179] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:40:50,191] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:40:50,196] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:50,196] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:50,905] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:50,906] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:40:51,012] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:40:51,013] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:40:51,013] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:40:51,070] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:40:51,105] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:40:51,106] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:40:51,676] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:40:51,719] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:40:51,763] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:40:51,766] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-07-01 15:40:51,773] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:51,773] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:52,460] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:52,465] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:52,465] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:53,402] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:53,410] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:53,410] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:54,551] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:40:54,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:40:54,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:40:55,723] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:02,702] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:41:02,702] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.55495619069255, 'mem_free': 13162233856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:41:02,703] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:41:02,703] - [basetestcase:467] INFO - Time to execute basesetup : 22.015109539031982
[2022-07-01 15:41:02,707] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:41:02,708] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:41:02,713] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:41:02,713] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:41:02,720] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:41:02,720] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:41:02,726] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:41:02,726] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:41:02,730] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:02,743] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:41:02,743] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:41:02,743] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:41:07,755] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:41:07,759] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:07,759] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:08,415] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:09,608] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:41:09,632] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:41:12,152] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:41:12,183] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:12,184] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:41:12,184] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:41:42,205] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:41:42,211] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:41:42,214] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:41:42,234] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.42315ms
[2022-07-01 15:41:42,234] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e3ea9527-41ba-4e3c-b374-9cdf1b18ef67', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.42315ms', 'executionTime': '18.347375ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:41:42,234] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:41:42,236] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:41:42,239] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:41:42,600] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 359.34814ms
[2022-07-01 15:41:42,600] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:41:42,608] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:41:42,612] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:41:42,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.989607ms
[2022-07-01 15:41:42,645] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:42,648] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:41:42,672] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:41:42,672] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:41:42,694] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:41:42,708] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:43,645] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:41:43,647] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:41:43,677] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.284275ms
[2022-07-01 15:41:43,677] - [base_gsi:282] INFO - BUILD INDEX on default(`employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr`) USING GSI
[2022-07-01 15:41:44,683] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr`) USING GSI
[2022-07-01 15:41:44,685] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%60%29+USING+GSI
[2022-07-01 15:41:44,711] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.982061ms
[2022-07-01 15:41:44,719] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr'
[2022-07-01 15:41:44,722] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%27
[2022-07-01 15:41:44,733] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.320445ms
[2022-07-01 15:41:45,739] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr'
[2022-07-01 15:41:45,741] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%27
[2022-07-01 15:41:45,750] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.731341ms
[2022-07-01 15:41:46,756] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr'
[2022-07-01 15:41:46,758] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%27
[2022-07-01 15:41:46,765] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.838325ms
[2022-07-01 15:41:47,771] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:41:47,774] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 15:41:47,778] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.881581ms
[2022-07-01 15:41:47,778] - [base_gsi:504] INFO - {'requestID': 'e1e4b677-a5b9-4dca-bc16-e4309d0a2a6d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr', 'index_id': 'c3524b32b4c02813', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.881581ms', 'executionTime': '2.81269ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 15:41:47,779] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:41:47,779] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:47,779] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:41:47,780] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:41:47,780] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:47,780] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:47,780] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:47,780] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:47,842] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:47,856] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:41:47,858] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 15:41:47,965] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 93.701979ms
[2022-07-01 15:41:47,965] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:41:47,965] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:41:49,122] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-07-01 15:41:49,152] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-07-01 15:41:49,157] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:41:49,160] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 15:41:49,163] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.044983ms
[2022-07-01 15:41:49,163] - [base_gsi:504] INFO - {'requestID': '9676be24-75d8-4ab2-b54b-a54708a5b462', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr', 'index_id': 'c3524b32b4c02813', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.044983ms', 'executionTime': '1.965712ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 15:41:49,164] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:41:49,164] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:49,164] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:41:49,164] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:41:49,165] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:49,165] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:49,165] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:49,165] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:41:49,221] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:41:49,235] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:41:49,237] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 15:41:49,328] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 81.73158ms
[2022-07-01 15:41:49,328] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:41:49,328] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:41:50,485] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr ON default USING GSI
[2022-07-01 15:41:50,487] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr+ON+default+USING+GSI
[2022-07-01 15:41:50,530] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.282285ms
[2022-07-01 15:41:50,538] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr'
[2022-07-01 15:41:50,541] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee116adbb2e1714a0a8b07a5fac62f93aajoin_yr%27
[2022-07-01 15:41:50,548] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.138681ms
[2022-07-01 15:41:50,549] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'eecd1c01-e4ef-4ee2-9cd1-e678d26aeccf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.138681ms', 'executionTime': '6.070058ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:41:50,558] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:50,562] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:50,562] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:51,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:51,356] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:41:51,356] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:41:51,478] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:41:51,485] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:51,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:51,489] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:52,216] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:52,223] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:41:52,224] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:41:52,340] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:41:52,348] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:41:52,348] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:41:52,348] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:41:52,350] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:41:52,355] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:41:52,365] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.944256ms
[2022-07-01 15:41:52,368] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:41:52,370] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:41:52,428] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.88452ms
[2022-07-01 15:41:52,439] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:41:52,439] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.63889474159468, 'mem_free': 12983230464, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:41:52,440] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:41:52,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:52,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:53,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:53,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:53,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:53,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:53,913] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:53,914] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:55,087] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:41:55,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:41:55,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:41:56,328] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:04,294] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-07-01 15:42:04,318] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:42:05,549] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:42:05,552] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:42:05,552] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:42:05,559] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:42:05,565] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:42:05,572] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:42:05,573] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:42:05,582] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:42:05,584] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:42:05,586] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:05,597] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:42:05,598] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:05,601] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:42:05,604] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:42:05,604] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:05,610] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:42:05,615] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:42:05,615] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:05,619] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:42:05,621] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:42:05,622] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:05,625] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:42:05,625] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-07-01 15:42:05,626] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:42:05,626] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 84.943s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_11'}
[2022-07-01 15:42:05,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:05,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:06,379] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:06,385] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:42:06,395] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:42:06,395] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-07-01 15:42:06,395] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:42:06,667] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:42:06,670] - [task:161] INFO -  {'uptime': '872', 'memoryTotal': 15466930176, 'memoryFree': 13143552000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:42:06,672] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:42:06,672] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:42:06,673] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:42:06,681] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:42:06,692] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:42:06,692] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:42:06,696] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:42:06,697] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:42:06,700] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:42:06,701] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:42:06,703] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:42:06,708] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:06,708] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:07,430] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:07,432] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:42:07,542] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:42:07,543] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:42:07,548] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:42:07,549] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:42:07,572] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:42:07,575] - [task:161] INFO -  {'uptime': '871', 'memoryTotal': 15466930176, 'memoryFree': 13143101440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:42:07,578] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:42:07,583] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:42:07,583] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:42:07,586] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:42:07,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:07,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:08,294] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:08,295] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:42:08,404] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:42:08,405] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:42:08,410] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:42:08,411] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:42:08,432] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:42:08,435] - [task:161] INFO -  {'uptime': '866', 'memoryTotal': 15466930176, 'memoryFree': 13143584768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:42:08,438] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:42:08,444] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:42:08,444] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:42:08,447] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:42:08,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:08,453] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:09,178] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:09,179] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:42:09,286] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:42:09,288] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:42:09,293] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:42:09,294] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:42:09,316] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:42:09,320] - [task:161] INFO -  {'uptime': '866', 'memoryTotal': 15466930176, 'memoryFree': 13143810048, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:42:09,323] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:42:09,327] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:42:09,327] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:42:09,331] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:42:09,335] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:09,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:10,045] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:10,047] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:42:10,156] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:42:10,157] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:42:10,163] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:42:10,163] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:42:10,181] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:42:10,289] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:42:15,293] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:42:15,307] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:42:15,313] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:15,314] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:16,018] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:16,019] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:42:16,130] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:42:16,132] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:42:16,132] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:42:16,199] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:42:16,232] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:42:16,233] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:42:16,797] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:42:16,836] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:42:17,511] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:42:17,514] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 15:42:17,520] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:17,520] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:18,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:18,264] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:18,264] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:19,332] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:19,348] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:19,348] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:20,565] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:20,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:20,577] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:21,768] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:28,795] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:42:28,795] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.45307270933218, 'mem_free': 13138812928, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:42:28,795] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:42:28,796] - [basetestcase:467] INFO - Time to execute basesetup : 23.136070489883423
[2022-07-01 15:42:28,800] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:42:28,800] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:42:28,807] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:42:28,807] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:42:28,812] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:42:28,813] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:42:28,819] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:42:28,819] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:42:28,824] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:42:28,836] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:42:28,836] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:42:28,836] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:42:33,846] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:42:33,849] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:42:33,849] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:42:34,556] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:42:35,785] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:42:35,811] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:42:38,329] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:42:38,358] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:42:38,359] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:42:38,359] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:43:08,390] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:43:08,396] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:43:08,398] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:43:08,418] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.541619ms
[2022-07-01 15:43:08,418] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b76548cc-c2cb-485a-a405-e3feb81ff6a1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.541619ms', 'executionTime': '18.468309ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:43:08,419] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:43:08,421] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:43:08,423] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:43:08,770] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 345.061648ms
[2022-07-01 15:43:08,771] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:43:08,778] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:43:08,781] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:43:08,793] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.236853ms
[2022-07-01 15:43:08,822] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:08,826] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:43:08,843] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:43:08,844] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:43:08,863] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:43:08,872] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:09,821] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee92da255d28164077942a0c7c807d126ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 15:43:09,823] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee92da255d28164077942a0c7c807d126ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 15:43:09,854] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.617793ms
[2022-07-01 15:43:09,854] - [base_gsi:282] INFO - BUILD INDEX on default(`employee92da255d28164077942a0c7c807d126ajoin_yr`) USING GSI
[2022-07-01 15:43:10,861] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee92da255d28164077942a0c7c807d126ajoin_yr`) USING GSI
[2022-07-01 15:43:10,863] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee92da255d28164077942a0c7c807d126ajoin_yr%60%29+USING+GSI
[2022-07-01 15:43:10,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 29.239223ms
[2022-07-01 15:43:10,902] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee92da255d28164077942a0c7c807d126ajoin_yr'
[2022-07-01 15:43:10,906] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee92da255d28164077942a0c7c807d126ajoin_yr%27
[2022-07-01 15:43:10,920] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 12.485787ms
[2022-07-01 15:43:11,927] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee92da255d28164077942a0c7c807d126ajoin_yr'
[2022-07-01 15:43:11,929] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee92da255d28164077942a0c7c807d126ajoin_yr%27
[2022-07-01 15:43:11,936] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.50782ms
[2022-07-01 15:43:12,943] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee92da255d28164077942a0c7c807d126ajoin_yr'
[2022-07-01 15:43:12,946] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee92da255d28164077942a0c7c807d126ajoin_yr%27
[2022-07-01 15:43:12,954] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.530647ms
[2022-07-01 15:43:13,960] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:43:13,963] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 15:43:13,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.722042ms
[2022-07-01 15:43:13,967] - [base_gsi:504] INFO - {'requestID': '14cf05fe-b9ac-4416-9764-794f0e2cd8a5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee92da255d28164077942a0c7c807d126ajoin_yr', 'index_id': '81d39334119285f4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.722042ms', 'executionTime': '2.654372ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 15:43:13,968] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 15:43:13,968] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:43:13,968] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 15:43:13,969] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 15:43:13,969] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:43:13,969] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:43:13,969] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:43:13,969] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 15:43:14,032] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 15:43:14,046] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 15:43:14,048] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 15:43:14,153] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 92.026254ms
[2022-07-01 15:43:14,153] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 15:43:14,154] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 15:43:15,638] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-01 15:43:18,027] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:43:18,058] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:43:18,059] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:43:18,615] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:43:18,657] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:43:18,700] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:43:18,703] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-01 15:43:20,708] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:43:20,708] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:20,712] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:43:20,716] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:43:20,716] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:20,719] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:43:20,723] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:43:20,723] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:20,726] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:43:20,731] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:43:20,731] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:20,734] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:43:20,739] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:20,741] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee92da255d28164077942a0c7c807d126ajoin_yr'
[2022-07-01 15:43:20,743] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee92da255d28164077942a0c7c807d126ajoin_yr%27
[2022-07-01 15:43:20,761] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 16.343702ms
[2022-07-01 15:43:20,761] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '426e5e78-3114-49b7-98c8-d90aa978c488', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '16.343702ms', 'executionTime': '16.260293ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:43:20,763] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee92da255d28164077942a0c7c807d126ajoin_yr'
[2022-07-01 15:43:20,766] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee92da255d28164077942a0c7c807d126ajoin_yr%27
[2022-07-01 15:43:20,768] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 973.401µs
[2022-07-01 15:43:20,768] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '110cfb08-ab8c-4d09-b571-ec28f8a86d00', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '973.401µs', 'executionTime': '927.626µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:43:20,772] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:20,782] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:20,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:20,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:21,551] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:21,559] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:43:21,559] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:43:21,678] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:43:21,686] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:21,688] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:21,689] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:22,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:22,484] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:43:22,485] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:43:22,605] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:43:22,613] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:22,613] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:43:22,613] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:43:22,615] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:43:22,618] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:43:22,628] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.728086ms
[2022-07-01 15:43:22,628] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b12a9d07-f7c4-4163-82d4-90e942a82f0a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.728086ms', 'executionTime': '8.658212ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:43:22,629] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:43:22,631] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:43:22,633] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:43:22,636] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.031645ms
[2022-07-01 15:43:22,636] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '29f8d562-2143-4e7e-9763-e706f285235f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.031645ms', 'executionTime': '982.378µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:43:22,640] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:43:22,640] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.29852217201936, 'mem_free': 12966416384, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:43:22,640] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:43:22,644] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:22,644] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:23,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:23,399] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:23,399] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:24,372] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:24,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:24,384] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:25,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:25,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:25,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:26,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:34,484] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 15:43:34,507] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:43:34,787] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:43:34,790] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:43:34,790] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:43:34,796] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:43:34,802] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:43:34,809] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:43:34,810] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:43:34,818] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:43:34,819] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:43:34,822] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:34,834] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:43:34,834] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:34,838] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:43:34,841] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:43:34,841] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:34,844] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:43:34,846] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:43:34,847] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:34,850] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:43:34,852] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:43:34,852] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:34,855] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:43:34,855] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 15:43:34,855] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:43:34,856] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 89.202s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_15-27-30/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-07-01 15:43:34,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:34,868] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:35,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:35,588] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:43:35,606] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 15:43:35,607] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-07-01 15:43:35,608] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 15:43:35,871] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 15:43:35,875] - [task:161] INFO -  {'uptime': '961', 'memoryTotal': 15466930176, 'memoryFree': 13116260352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:43:35,877] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 15:43:35,877] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 15:43:35,878] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 15:43:35,885] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 15:43:35,896] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 15:43:35,896] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 15:43:35,898] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 15:43:35,900] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 15:43:35,900] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 15:43:35,900] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 15:43:35,903] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:43:35,906] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:35,907] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:36,613] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:36,614] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:43:36,721] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:43:36,723] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:43:36,729] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:43:36,729] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:43:36,756] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 15:43:36,759] - [task:161] INFO -  {'uptime': '956', 'memoryTotal': 15466930176, 'memoryFree': 13116305408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:43:36,761] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:43:36,766] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 15:43:36,766] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 15:43:36,769] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:43:36,773] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:36,773] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:37,508] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:37,509] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:43:37,615] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:43:37,616] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:43:37,621] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:43:37,622] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:43:37,643] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 15:43:37,646] - [task:161] INFO -  {'uptime': '956', 'memoryTotal': 15466930176, 'memoryFree': 13115994112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:43:37,649] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:43:37,653] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 15:43:37,653] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 15:43:37,656] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:43:37,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:37,660] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:38,365] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:38,366] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:43:38,475] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:43:38,476] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:43:38,481] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:43:38,482] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:43:38,501] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 15:43:38,504] - [task:161] INFO -  {'uptime': '956', 'memoryTotal': 15466930176, 'memoryFree': 13115375616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 15:43:38,506] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 15:43:38,510] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 15:43:38,511] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 15:43:38,513] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 15:43:38,518] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:38,518] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:39,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:39,234] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:43:39,341] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:43:39,342] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:43:39,346] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 15:43:39,347] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-07-01 15:43:39,368] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 15:43:39,477] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:43:44,481] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 15:43:44,494] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 15:43:44,500] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:44,500] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:45,245] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:45,246] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 15:43:45,357] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 15:43:45,359] - [remote_util:5249] INFO - b'ok'
[2022-07-01 15:43:45,360] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 15:43:45,383] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 15:43:45,414] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 15:43:45,414] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 15:43:45,974] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:43:46,011] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:43:46,686] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 15:43:46,689] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-07-01 15:43:46,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:46,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:47,471] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:47,475] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:47,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:48,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:48,599] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:48,599] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:49,875] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:49,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:43:49,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:43:51,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:43:58,552] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:43:58,553] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.25168550202744, 'mem_free': 13113040896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:43:58,553] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:43:58,553] - [basetestcase:467] INFO - Time to execute basesetup : 23.688457250595093
[2022-07-01 15:43:58,558] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:43:58,558] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:43:58,564] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:43:58,564] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:43:58,569] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:43:58,569] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:43:58,575] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 15:43:58,575] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 15:43:58,579] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:43:58,592] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-07-01 15:43:58,593] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 15:43:58,593] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 15:44:03,604] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 15:44:03,607] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:03,607] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:04,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:05,550] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 15:44:05,576] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 15:44:08,081] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 15:44:08,113] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:08,114] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 15:44:08,114] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 15:44:38,143] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 15:44:38,149] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:44:38,151] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:44:38,170] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.572109ms
[2022-07-01 15:44:38,171] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '17bd698c-c665-4716-8d6f-f61e8db42acf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '17.572109ms', 'executionTime': '17.482672ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 15:44:38,171] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 15:44:38,173] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 15:44:38,175] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 15:44:38,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 382.966095ms
[2022-07-01 15:44:38,560] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 15:44:38,569] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:44:38,573] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:44:38,587] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.37624ms
[2022-07-01 15:44:38,620] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:38,623] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 15:44:38,642] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 15:44:38,642] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 15:44:38,659] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 15:44:38,673] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:38,690] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:38,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:38,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:39,571] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:39,581] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:44:39,581] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 15:44:39,707] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:44:39,714] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:39,717] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:39,717] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:40,565] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:40,572] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 15:44:40,573] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 15:44:40,705] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 15:44:40,716] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 15:44:40,716] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 15:44:40,716] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:44:40,718] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 15:44:40,721] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 15:44:40,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.752336ms
[2022-07-01 15:44:40,732] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 15:44:40,734] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 15:44:40,786] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.014904ms
[2022-07-01 15:44:40,800] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 15:44:40,801] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.489971075442527, 'mem_free': 12939632640, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 15:44:40,802] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 15:44:40,808] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:40,809] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:41,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:41,620] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:41,620] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:42,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:42,821] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:42,822] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:44,193] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:44,203] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 15:44:44,203] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 15:44:45,582] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 15:44:53,622] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-07-01 15:44:53,647] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 15:44:54,838] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 15:44:54,842] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 15:44:54,842] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 15:44:54,848] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:44:54,855] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:44:54,861] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 15:44:54,862] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 15:44:54,871] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 15:44:54,872] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 15:44:54,874] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:44:54,887] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 15:44:54,888] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:44:54,891] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 15:44:54,898] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 15:44:54,898] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:44:54,903] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 15:44:54,905] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 15:44:54,905] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:44:54,909] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 15:44:54,913] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 15:44:54,913] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 15:44:54,917] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 15:44:54,917] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-07-01 15:44:54,917] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 15:44:54,918] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 80.095s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 13231 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

Binary file /opt/build/test.log matches
Version: versions-01.07.2022-14.05.cfg
Build Log: make-01.07.2022-14.05.log
Server Log: logs-01.07.2022-14.05.tar.gz

Finished