Building

Started building at 2022/07/01 11:17:25
Using pegged server, 1432 build
Calculating base
Updating mirror
Basing run on 7.2.0-1432 c44d315bd
Updating tree for run 01.07.2022-11.17
query is at f979279, changes since last good build: none
gometa is at 8625ed1, changes since last good build: 

ns_server is at c9545cf, changes since last good build: none
couchstore is at f0c688c, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at d419a87, changes since last good build: none
Switching indexing to unstable
indexing is at b49b9d4, changes since last good build: none
Switching plasma to unstable
plasma is at 85217ad, changes since last good build: 
fatal: Invalid revision range 67b39ac64e16632744752cec29ead6c6bfd8d61e..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at a237ed7, changes since last good build: none
Switching testrunner to master
testrunner is at 6e16c8a, changes since last good build: none
Pulling in uncommitted change 175819 at refs/changes/19/175819/4
Total 5 (delta 2), reused 3 (delta 2)
[unstable 83cc1b88] MB-51825: kvsender - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 7 15:41:01 2022 +0530
 1 file changed, 42 insertions(+), 24 deletions(-)
Pulling in uncommitted change 176419 at refs/changes/19/176419/2
Total 18 (delta 12), reused 15 (delta 12)
[unstable dbb3dd0a] MB-51825: Add numVBuckets to index instance
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 12:52:17 2022 +0530
 3 files changed, 27 insertions(+), 3 deletions(-)
Pulling in uncommitted change 176420 at refs/changes/20/176420/4
Total 34 (delta 24), reused 29 (delta 24)
[unstable c8739ce2] MB-51825: Use numVBuckets from index inst in storage_managers
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 13:05:54 2022 +0530
 2 files changed, 16 insertions(+), 18 deletions(-)
Pulling in uncommitted change 176421 at refs/changes/21/176421/3
Total 40 (delta 28), reused 34 (delta 28)
[unstable a420334f] MB-51825: Use numVBuckets from instance object in scan_coordinator
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 13:08:59 2022 +0530
 2 files changed, 3 insertions(+), 3 deletions(-)
Pulling in uncommitted change 176426 at refs/changes/26/176426/2
[unstable 6c697cd9] MB-51825: Dont use numVBuckets from config in indexer and NewSlice
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 14:27:27 2022 +0530
 5 files changed, 109 insertions(+), 42 deletions(-)
Pulling in uncommitted change 176427 at refs/changes/27/176427/2
Total 46 (delta 35), reused 42 (delta 35)
[unstable ffb66bda] MB-51825: Use NumVBuckets from index inst in MutationMgr &  StreamReader
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 14:33:49 2022 +0530
 2 files changed, 16 insertions(+), 14 deletions(-)
Pulling in uncommitted change 176523 at refs/changes/23/176523/2
Total 56 (delta 42), reused 51 (delta 42)
[unstable 06aa5346] MB-51825: Fix memedb_slice_test adding numVBuckets arg
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Jun 22 16:31:49 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 176730 at refs/changes/30/176730/1
Total 5 (delta 4), reused 5 (delta 4)
[unstable 45334b37] MB-52151: Dont meter deletes in insert paths
 Author: Sai Krishna Teja Kommaraju 
 Date: Mon Jun 27 20:11:44 2022 +0530
 1 file changed, 11 insertions(+), 8 deletions(-)
Pulling in uncommitted change 176947 at refs/changes/47/176947/1
Total 20 (delta 17), reused 20 (delta 17)
[unstable e39c70d0] MB-52308 Part 3 (7.2.0 1517) Elixir Pause-Resume feature for GSI
 Author: cherkauer-couchbase 
 Date: Thu Jun 30 17:33:44 2022 -0700
 9 files changed, 579 insertions(+), 335 deletions(-)
 create mode 100644 secondary/indexer/sched_token_monitor.go
Pulling in uncommitted change 176873 at refs/changes/73/176873/2
Total 3 (delta 0), reused 0 (delta 0)
[unstable d44f30b] MB-100: Fix addInstance for dedicated LSS cleanup
 Author: saptarshi.sen 
 Date: Wed Jun 29 11:07:35 2022 -0700
 1 file changed, 6 insertions(+)
Building cmakefiles and deps
Building main product

... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo


... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo

Testing

Started testing at 2022/07/01 11:50:27
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Jul 01 11:55:38 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 01 11:59:19 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jul 01 11:59:57 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Jul 01 12:01:06 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Jul 01 12:01:55 b" 'stop_on_failure': 'True'}"
Jul 01 12:01:55 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 01 12:01:55 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_1'}"
Jul 01 12:01:55 b'-->result: '
Jul 01 12:01:55 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Jul 01 12:01:55 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jul 01 12:01:55 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_2'}"
Jul 01 12:01:55 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:01:55 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Jul 01 12:01:55 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_3'}"
Jul 01 12:01:55 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:01:55 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:01:55 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Jul 01 12:01:55 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_4'}"
Jul 01 12:01:55 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:01:55 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:01:55 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 12:11:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Jul 01 12:12:34 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Jul 01 12:15:58 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 12:19:34 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 12:22:06 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Jul 01 12:22:06 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_5'}"
Jul 01 12:22:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Jul 01 12:22:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Jul 01 12:22:06 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_6'}"
Jul 01 12:22:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 12:22:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Jul 01 12:22:06 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_7'}"
Jul 01 12:22:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Jul 01 12:22:06 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Jul 01 12:22:06 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_11-50-48/test_8'}"
Jul 01 12:22:06 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jul 01 12:22:06 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jul 01 12:22:06 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Jul 01 12:22:06 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Jul 01 12:22:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 01 12:22:07 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jul 01 12:22:07 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Jul 01 12:22:07 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Jul 01 12:22:07 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Jul 01 12:22:07 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Jul 01 12:22:07 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Jul 01 12:22:07 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.02s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000 items took 15.758035ms -> 1.0153550236434937e+06 items/s conflicts 1
--- PASS: TestInsertPerf (0.02s)
=== RUN   TestGetPerf
16000 items took 7.674697ms -> 2.0847728581336825e+06 items/s
--- PASS: TestGetPerf (0.01s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3450,
"memory_used":            45520,
"node_allocs":            1000,
"node_frees":             0,
"level_node_distribution":{
"level0": 747,
"level1": 181,
"level2": 56,
"level3": 13,
"level4": 2,
"level5": 1,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [105 161 346 379 434 523 713]
No of items in each range [105 56 185 33 55 89 190 287]
--- PASS: TestGetRangeSplitItems (0.00s)
=== RUN   TestBuilder
{
"node_count":             50000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3368,
"memory_used":            2269408,
"node_allocs":            50000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37380,
"level1": 9466,
"level2": 2370,
"level3": 578,
"level4": 152,
"level5": 40,
"level6": 9,
"level7": 4,
"level8": 1,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 7.466969ms to build 50000 items, 6.6961575e+06 items/sec
Took 2.962931ms to iterate 50000 items
--- PASS: TestBuilder (0.01s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/nitro/skiplist	0.071s
=== RUN   TestZstdSimple
--- PASS: TestZstdSimple (0.00s)
=== RUN   TestZstdCompressBound
--- PASS: TestZstdCompressBound (3.08s)
=== RUN   TestZstdErrors
--- PASS: TestZstdErrors (0.00s)
=== RUN   TestZstdCompressLevels
--- PASS: TestZstdCompressLevels (0.74s)
=== RUN   TestZstdEmptySrc
--- PASS: TestZstdEmptySrc (0.00s)
=== RUN   TestZstdLargeSrc
--- PASS: TestZstdLargeSrc (0.00s)
PASS
ok  	github.com/couchbase/plasma/zstd	3.831s
=== RUN   TestAutoTunerWriteUsageStats
--- PASS: TestAutoTunerWriteUsageStats (10.05s)
=== RUN   TestAutoTunerReadUsageStats
--- PASS: TestAutoTunerReadUsageStats (7.75s)
=== RUN   TestAutoTunerCleanerUsageStats
--- PASS: TestAutoTunerCleanerUsageStats (8.66s)
=== RUN   TestAutoTunerDiskStats
--- PASS: TestAutoTunerDiskStats (2.50s)
=== RUN   TestAutoTunerTargetFragRatio
--- PASS: TestAutoTunerTargetFragRatio (0.00s)
=== RUN   TestAutoTunerExcessUsedSpace
--- PASS: TestAutoTunerExcessUsedSpace (0.00s)
=== RUN   TestAutoTunerUsedSpaceRatio
--- PASS: TestAutoTunerUsedSpaceRatio (0.00s)
=== RUN   TestAutoTunerAdjustFragRatio
--- PASS: TestAutoTunerAdjustFragRatio (0.00s)
=== RUN   TestAutoTuneFlushBufferAdjustMemQuotaSingleShard
--- PASS: TestAutoTuneFlushBufferAdjustMemQuotaSingleShard (169.36s)
=== RUN   TestAutoTuneFlushBufferAdjustMemQuotaManyShards
--- PASS: TestAutoTuneFlushBufferAdjustMemQuotaManyShards (180.80s)
=== RUN   TestAutoTuneFlushBufferRebalanceIdleShards
--- PASS: TestAutoTuneFlushBufferRebalanceIdleShards (172.79s)
=== RUN   TestAutoTuneFlushBufferGetUsedMemory
--- PASS: TestAutoTuneFlushBufferGetUsedMemory (118.37s)
=== RUN   TestBloom
--- PASS: TestBloom (5.19s)
=== RUN   TestBloomDisableEnable
--- PASS: TestBloomDisableEnable (4.06s)
=== RUN   TestBloomDisable
--- PASS: TestBloomDisable (0.11s)
=== RUN   TestBloomFreeDuringLookup
--- PASS: TestBloomFreeDuringLookup (0.10s)
=== RUN   TestBloomRecoveryFreeDuringLookup
--- PASS: TestBloomRecoveryFreeDuringLookup (0.14s)
=== RUN   TestBloomRecoverySwapInLookup
--- PASS: TestBloomRecoverySwapInLookup (0.16s)
=== RUN   TestBloomRecoverySwapOutLookup
--- PASS: TestBloomRecoverySwapOutLookup (0.15s)
=== RUN   TestBloomRecoveryInserts
--- PASS: TestBloomRecoveryInserts (0.13s)
=== RUN   TestBloomRecovery
--- PASS: TestBloomRecovery (0.18s)
=== RUN   TestBloomStats
--- PASS: TestBloomStats (4.07s)
=== RUN   TestBloomStatsRecovery
--- PASS: TestBloomStatsRecovery (0.86s)
=== RUN   TestBloomFilterSimple
--- PASS: TestBloomFilterSimple (0.00s)
=== RUN   TestBloomFilterConcurrent
--- PASS: TestBloomFilterConcurrent (22.16s)
=== RUN   TestBitArrayConcurrent
--- PASS: TestBitArrayConcurrent (0.69s)
=== RUN   TestBloomCapacity
--- PASS: TestBloomCapacity (0.00s)
=== RUN   TestBloomNumHashFuncs
--- PASS: TestBloomNumHashFuncs (0.00s)
=== RUN   TestBloomTestAndAdd
--- PASS: TestBloomTestAndAdd (0.23s)
=== RUN   TestBloomReset
--- PASS: TestBloomReset (0.00s)
=== RUN   TestDiag
--- PASS: TestDiag (0.71s)
=== RUN   TestDumpLog
--- PASS: TestDumpLog (0.12s)
=== RUN   TestExtrasN1
=== RUN   TestExtrasN2
=== RUN   TestExtrasN3
=== RUN   TestGMRecovery
--- PASS: TestGMRecovery (9.29s)
=== RUN   TestIteratorSimple
--- PASS: TestIteratorSimple (5.38s)
=== RUN   TestIteratorSeek
--- PASS: TestIteratorSeek (6.57s)
=== RUN   TestPlasmaIteratorSeekFirst
--- PASS: TestPlasmaIteratorSeekFirst (0.59s)
=== RUN   TestPlasmaIteratorSwapin
--- PASS: TestPlasmaIteratorSwapin (5.53s)
=== RUN   TestIteratorSetEnd
--- PASS: TestIteratorSetEnd (0.80s)
=== RUN   TestIterHiItm
--- PASS: TestIterHiItm (2.03s)
=== RUN   TestIterDeleteSplitMerge
--- PASS: TestIterDeleteSplitMerge (0.09s)
=== RUN   TestKeySamplingSingle
--- PASS: TestKeySamplingSingle (0.17s)
=== RUN   TestKeySamplingAll
--- PASS: TestKeySamplingAll (0.17s)
=== RUN   TestKeySamplingEmpty
--- PASS: TestKeySamplingEmpty (0.11s)
=== RUN   TestKeySamplingExceed
--- PASS: TestKeySamplingExceed (0.16s)
=== RUN   TestLogOperation
--- PASS: TestLogOperation (61.12s)
=== RUN   TestLogLargeSize
--- PASS: TestLogLargeSize (0.20s)
=== RUN   TestLogTrim
--- PASS: TestLogTrim (61.14s)
=== RUN   TestLogSuperblockCorruption
--- PASS: TestLogSuperblockCorruption (61.01s)
=== RUN   TestLogTrimHolePunch
--- PASS: TestLogTrimHolePunch (50.42s)
=== RUN   TestLogMissingAndTruncatedSegments
--- PASS: TestLogMissingAndTruncatedSegments (0.08s)
=== RUN   TestLogReadBeyondMaxFileIndex
--- PASS: TestLogReadBeyondMaxFileIndex (2.68s)
=== RUN   TestLogReadEOFWithMMap
--- PASS: TestLogReadEOFWithMMap (0.01s)
=== RUN   TestShardLSSCleaning
--- PASS: TestShardLSSCleaning (0.27s)
=== RUN   TestShardLSSCleaningDeleteInstance
--- PASS: TestShardLSSCleaningDeleteInstance (0.25s)
=== RUN   TestShardLSSCleaningCorruptInstance
--- PASS: TestShardLSSCleaningCorruptInstance (0.23s)
=== RUN   TestPlasmaLSSCleaner
--- PASS: TestPlasmaLSSCleaner (224.10s)
=== RUN   TestLSSBasic
--- PASS: TestLSSBasic (0.08s)
=== RUN   TestLSSConcurrent
--- PASS: TestLSSConcurrent (0.90s)
=== RUN   TestLSSCleaner
--- PASS: TestLSSCleaner (13.34s)
=== RUN   TestLSSSuperBlock
--- PASS: TestLSSSuperBlock (1.17s)
=== RUN   TestLSSLargeSinglePayload
--- PASS: TestLSSLargeSinglePayload (0.87s)
=== RUN   TestLSSUnstableEnvironment
--- PASS: TestLSSUnstableEnvironment (10.27s)
=== RUN   TestLSSSmallFlushBuffer
--- PASS: TestLSSSmallFlushBuffer (0.01s)
=== RUN   TestLSSTrimFlushBufferGC
--- PASS: TestLSSTrimFlushBufferGC (1.94s)
=== RUN   TestLSSTrimFlushBufferNoIO
--- PASS: TestLSSTrimFlushBufferNoIO (30.13s)
=== RUN   TestLSSTrimFlushBufferWithIO
--- PASS: TestLSSTrimFlushBufferWithIO (33.55s)
=== RUN   TestLSSExtendFlushBufferWithIO
--- PASS: TestLSSExtendFlushBufferWithIO (30.02s)
=== RUN   TestLSSCtxTrimFlushBuffer
--- PASS: TestLSSCtxTrimFlushBuffer (4.26s)
=== RUN   TestMem
Plasma: Adaptive memory quota tuning (decrementing): RSS:1629806592, freePercent:84.78760924613874, currentQuota=1099511627776, newQuota=1073741824, netGrowth=0, percent=99Plasma: Adaptive memory quota tuning (incrementing): RSS:1629691904, freePercent: 84.78853612689898, currentQuota=0, newQuota=10995116277--- PASS: TestMem (15.10s)
=== RUN   TestCpu
--- PASS: TestCpu (15.22s)
=== RUN   TestTopTen20
--- PASS: TestTopTen20 (0.74s)
=== RUN   TestTopTen5
--- PASS: TestTopTen5 (0.24s)
=== RUN   TestMVCCSimple
--- PASS: TestMVCCSimple (0.28s)
=== RUN   TestMVCCLookup
--- PASS: TestMVCCLookup (0.19s)
=== RUN   TestMVCCIteratorRefresh
--- PASS: TestMVCCIteratorRefresh (4.93s)
=== RUN   TestMVCCIteratorRefreshEveryRow
--- PASS: TestMVCCIteratorRefreshEveryRow (0.88s)
=== RUN   TestMVCCGarbageCollection
--- PASS: TestMVCCGarbageCollection (0.17s)
=== RUN   TestMVCCRecoveryPoint
--- PASS: TestMVCCRecoveryPoint (1.98s)
=== RUN   TestMVCCRollbackMergeSibling
--- PASS: TestMVCCRollbackMergeSibling (0.14s)
=== RUN   TestMVCCRollbackCompact
--- PASS: TestMVCCRollbackCompact (0.13s)
=== RUN   TestMVCCRollbackSplit
--- PASS: TestMVCCRollbackSplit (0.17s)
=== RUN   TestMVCCRollbackItemsNotInSnapshot
--- PASS: TestMVCCRollbackItemsNotInSnapshot (0.21s)
=== RUN   TestMVCCRecoveryPointRollbackedSnapshot
--- PASS: TestMVCCRecoveryPointRollbackedSnapshot (0.97s)
=== RUN   TestMVCCRollbackBetweenRecoveryPoint
--- PASS: TestMVCCRollbackBetweenRecoveryPoint (0.98s)
=== RUN   TestMVCCRecoveryPointCrash
--- PASS: TestMVCCRecoveryPointCrash (0.17s)
=== RUN   TestMVCCIntervalGC
--- PASS: TestMVCCIntervalGC (0.28s)
=== RUN   TestMVCCItemsCount
--- PASS: TestMVCCItemsCount (0.42s)
=== RUN   TestLargeItems
--- PASS: TestLargeItems (112.35s)
=== RUN   TestTooLargeKey
--- PASS: TestTooLargeKey (3.51s)
=== RUN   TestMVCCItemUpdateSize
--- PASS: TestMVCCItemUpdateSize (0.31s)
=== RUN   TestEvictionStats
--- PASS: TestEvictionStats (0.50s)
=== RUN   TestReaderCacheStats
--- PASS: TestReaderCacheStats (1.22s)
=== RUN   TestInvalidSnapshot
--- PASS: TestInvalidSnapshot (0.95s)
=== RUN   TestEmptyKeyInsert
--- PASS: TestEmptyKeyInsert (0.13s)
=== RUN   TestMVCCRecoveryPointError
--- PASS: TestMVCCRecoveryPointError (0.13s)
=== RUN   TestMVCCReaderPurgeSequential
--- PASS: TestMVCCReaderPurgeSequential (0.28s)
=== RUN   TestMVCCReaderNoPurge
--- PASS: TestMVCCReaderNoPurge (0.28s)
=== RUN   TestMVCCReaderPurgeAfterUpdate
--- PASS: TestMVCCReaderPurgeAfterUpdate (0.29s)
=== RUN   TestMVCCReaderPurgeAfterRollback
--- PASS: TestMVCCReaderPurgeAfterRollback (0.31s)
=== RUN   TestMVCCReaderPurgeSimple
--- PASS: TestMVCCReaderPurgeSimple (0.17s)
=== RUN   TestMVCCReaderPurgeRandom
--- PASS: TestMVCCReaderPurgeRandom (0.28s)
=== RUN   TestMVCCReaderPurgePageFlag
--- PASS: TestMVCCReaderPurgePageFlag (0.19s)
=== RUN   TestMVCCPurgeRatioWithRollback
--- PASS: TestMVCCPurgeRatioWithRollback (15.91s)
=== RUN   TestComputeItemsCountMVCCWithRollbackI
--- PASS: TestComputeItemsCountMVCCWithRollbackI (0.21s)
=== RUN   TestComputeItemsCountMVCCWithRollbackII
--- PASS: TestComputeItemsCountMVCCWithRollbackII (0.15s)
=== RUN   TestComputeItemsCountMVCCWithRollbackIII
--- PASS: TestComputeItemsCountMVCCWithRollbackIII (0.14s)
=== RUN   TestComputeItemsCountMVCCWithRollbackIV
--- PASS: TestComputeItemsCountMVCCWithRollbackIV (0.21s)
=== RUN   TestMVCCPurgedRecordsWithCompactFullMarshalAndCascadedEmptyPagesMerge
--- PASS: TestMVCCPurgedRecordsWithCompactFullMarshalAndCascadedEmptyPagesMerge (1.89s)
=== RUN   TestMaxDeltaChainLenWithCascadedEmptyPagesMerge
--- PASS: TestMaxDeltaChainLenWithCascadedEmptyPagesMerge (1.56s)
=== RUN   TestAutoHoleCleaner
--- PASS: TestAutoHoleCleaner (34.76s)
=== RUN   TestAutoHoleCleaner5Indexes
--- PASS: TestAutoHoleCleaner5Indexes (203.99s)
=== RUN   TestIteratorReportedHoleRegionBoundary
--- PASS: TestIteratorReportedHoleRegionBoundary (0.24s)
=== RUN   TestFullRangeHoleScans
--- PASS: TestFullRangeHoleScans (0.42s)
=== RUN   TestOverlappingRangeHoleScans
--- PASS: TestOverlappingRangeHoleScans (0.43s)
=== RUN   TestMVCCIteratorSMRRefreshOnHoleScan
--- PASS: TestMVCCIteratorSMRRefreshOnHoleScan (8.29s)
=== RUN   TestAutoHoleCleanerWithRecovery
--- PASS: TestAutoHoleCleanerWithRecovery (3.03s)
=== RUN   TestPageMergeCorrectness2
--- PASS: TestPageMergeCorrectness2 (0.00s)
=== RUN   TestPageMergeCorrectness
--- PASS: TestPageMergeCorrectness (0.00s)
=== RUN   TestPageMarshalFull
--- PASS: TestPageMarshalFull (0.01s)
=== RUN   TestPageMergeMarshal
--- PASS: TestPageMergeMarshal (0.00s)
=== RUN   TestPageOperations
--- PASS: TestPageOperations (0.03s)
=== RUN   TestPageIterator
--- PASS: TestPageIterator (0.00s)
=== RUN   TestPageMarshal
--- PASS: TestPageMarshal (0.02s)
=== RUN   TestPageMergeCorrectness3
--- PASS: TestPageMergeCorrectness3 (0.00s)
=== RUN   TestPageHasDataRecords
--- PASS: TestPageHasDataRecords (0.00s)
=== RUN   TestPlasmaPageVisitor
--- PASS: TestPlasmaPageVisitor (4.95s)
=== RUN   TestPageRingVisitor
--- PASS: TestPageRingVisitor (4.78s)
=== RUN   TestPauseVisitorOnLowMemory
--- PASS: TestPauseVisitorOnLowMemory (1.21s)
=== RUN   TestCheckpointRecovery
--- PASS: TestCheckpointRecovery (8.98s)
=== RUN   TestPageCorruption
--- PASS: TestPageCorruption (0.93s)
=== RUN   TestCheckPointRecoveryFollowCleaning
--- PASS: TestCheckPointRecoveryFollowCleaning (0.16s)
=== RUN   TestFragmentationWithZeroItems
--- PASS: TestFragmentationWithZeroItems (1.24s)
=== RUN   TestEvictOnPersist
--- PASS: TestEvictOnPersist (0.22s)
=== RUN   TestPlasmaSimple
--- PASS: TestPlasmaSimple (14.63s)
=== RUN   TestPlasmaCompression
--- PASS: TestPlasmaCompression (0.10s)
=== RUN   TestPlasmaCompressionWrong
--- PASS: TestPlasmaCompressionWrong (0.12s)
=== RUN   TestPlasmaInMemCompression
--- PASS: TestPlasmaInMemCompression (0.11s)
=== RUN   TestPlasmaInMemCompressionZstd
--- PASS: TestPlasmaInMemCompressionZstd (0.12s)
=== RUN   TestPlasmaInMemCompressionWrong
--- PASS: TestPlasmaInMemCompressionWrong (0.12s)
=== RUN   TestSpoiledConfig
--- PASS: TestSpoiledConfig (0.12s)
=== RUN   TestPlasmaErrorFile
--- PASS: TestPlasmaErrorFile (0.11s)
=== RUN   TestPlasmaPersistor
--- PASS: TestPlasmaPersistor (11.12s)
=== RUN   TestPlasmaEvictionLSSDataSize
--- PASS: TestPlasmaEvictionLSSDataSize (0.12s)
=== RUN   TestPlasmaEviction
--- PASS: TestPlasmaEviction (31.16s)
=== RUN   TestConcurrDelOps
--- PASS: TestConcurrDelOps (70.45s)
=== RUN   TestPlasmaDataSize
--- PASS: TestPlasmaDataSize (0.11s)
=== RUN   TestLargeBasePage
--- PASS: TestLargeBasePage (53.82s)
=== RUN   TestLargeValue
--- PASS: TestLargeValue (105.21s)
=== RUN   TestPlasmaTooLargeKey
--- PASS: TestPlasmaTooLargeKey (3.49s)
=== RUN   TestEvictAfterMerge
--- PASS: TestEvictAfterMerge (0.19s)
=== RUN   TestEvictDirty
--- PASS: TestEvictDirty (0.22s)
=== RUN   TestEvictUnderQuota
--- PASS: TestEvictUnderQuota (60.19s)
=== RUN   TestEvictSetting
--- PASS: TestEvictSetting (1.26s)
=== RUN   TestBasePageAfterCompaction
--- PASS: TestBasePageAfterCompaction (0.21s)
=== RUN   TestSwapout
--- PASS: TestSwapout (0.11s)
=== RUN   TestSwapoutSplitBasePage
--- PASS: TestSwapoutSplitBasePage (0.11s)
=== RUN   TestCompactFullMarshal
--- PASS: TestCompactFullMarshal (0.17s)
=== RUN   TestPageStats
--- PASS: TestPageStats (2.36s)
=== RUN   TestPageStatsTinyIndex
--- PASS: TestPageStatsTinyIndex (0.25s)
=== RUN   TestPageStatsTinyIndexOnRecovery
--- PASS: TestPageStatsTinyIndexOnRecovery (0.16s)
=== RUN   TestPageStatsTinyIndexOnSplitAndMerge
--- PASS: TestPageStatsTinyIndexOnSplitAndMerge (0.15s)
=== RUN   TestPageCompress
--- PASS: TestPageCompress (0.13s)
=== RUN   TestPageCompressSwapin
--- PASS: TestPageCompressSwapin (0.13s)
=== RUN   TestPageCompressStats
--- PASS: TestPageCompressStats (0.74s)
=== RUN   TestPageDecompressStats
--- PASS: TestPageDecompressStats (0.12s)
=== RUN   TestSharedDedicatedDataSize
--- PASS: TestSharedDedicatedDataSize (4.00s)
=== RUN   TestLastRpSns
--- PASS: TestLastRpSns (0.16s)
=== RUN   TestPageCompressState
--- PASS: TestPageCompressState (0.12s)
=== RUN   TestPageCompressDuringBurst
--- PASS: TestPageCompressDuringBurst (0.12s)
=== RUN   TestPageDontDecompressDuringScan
--- PASS: TestPageDontDecompressDuringScan (0.19s)
=== RUN   TestPageDecompressAndCompressSwapin
--- PASS: TestPageDecompressAndCompressSwapin (2.27s)
=== RUN   TestPageCompressibleStat
--- PASS: TestPageCompressibleStat (0.63s)
=== RUN   TestPageCompressibleStatRecovery
--- PASS: TestPageCompressibleStatRecovery (0.27s)
=== RUN   TestPageCompressBeforeEvictPercent
--- PASS: TestPageCompressBeforeEvictPercent (0.80s)
=== RUN   TestPageCompressDecompressAfterDisable
--- PASS: TestPageCompressDecompressAfterDisable (0.78s)
=== RUN   TestWrittenDataSz
--- PASS: TestWrittenDataSz (3.45s)
=== RUN   TestWrittenDataSzAfterRecoveryCleaning
--- PASS: TestWrittenDataSzAfterRecoveryCleaning (4.06s)
=== RUN   TestWrittenHdrSz
--- PASS: TestWrittenHdrSz (3.15s)
=== RUN   TestPersistConfigUpgrade
--- PASS: TestPersistConfigUpgrade (0.09s)
=== RUN   TestLSSSegmentSize
--- PASS: TestLSSSegmentSize (0.27s)
=== RUN   TestPlasmaFlushBufferSzCfg
--- PASS: TestPlasmaFlushBufferSzCfg (0.22s)
=== RUN   TestCompactionCountwithCompactFullMarshal
--- PASS: TestCompactionCountwithCompactFullMarshal (0.16s)
=== RUN   TestCompactionCountwithCompactFullMarshalSMO
--- PASS: TestCompactionCountwithCompactFullMarshalSMO (0.13s)
=== RUN   TestPageHasDataRecordsOnCompactFullMarshal
--- PASS: TestPageHasDataRecordsOnCompactFullMarshal (0.18s)
=== RUN   TestPauseReaderOnLowMemory
--- PASS: TestPauseReaderOnLowMemory (1.12s)
=== RUN   TestRecoveryCleanerFragRatio
--- PASS: TestRecoveryCleanerFragRatio (225.12s)
=== RUN   TestRecoveryCleanerRelocation
--- PASS: TestRecoveryCleanerRelocation (225.14s)
=== RUN   TestRecoveryCleanerDataSize
--- PASS: TestRecoveryCleanerDataSize (224.97s)
=== RUN   TestRecoveryCleanerDeleteInstance
--- PASS: TestRecoveryCleanerDeleteInstance (448.79s)
=== RUN   TestRecoveryCleanerRecoveryPoint
--- PASS: TestRecoveryCleanerRecoveryPoint (28.48s)
=== RUN   TestRecoveryCleanerCorruptInstance
--- PASS: TestRecoveryCleanerCorruptInstance (0.28s)
=== RUN   TestRecoveryCleanerAhead
--- PASS: TestRecoveryCleanerAhead (4.41s)
=== RUN   TestRecoveryCleanerAheadAfterRecovery
--- PASS: TestRecoveryCleanerAheadAfterRecovery (2.31s)
=== RUN   TestCleaningUncommittedData
--- PASS: TestCleaningUncommittedData (0.11s)
=== RUN   TestPlasmaRecoverySimple
--- PASS: TestPlasmaRecoverySimple (0.12s)
=== RUN   TestPlasmaRecovery
--- PASS: TestPlasmaRecovery (32.36s)
=== RUN   TestShardRecoveryShared
--- PASS: TestShardRecoveryShared (11.64s)
=== RUN   TestShardRecoveryRecoveryLogAhead
--- PASS: TestShardRecoveryRecoveryLogAhead (34.49s)
=== RUN   TestShardRecoveryDataLogAhead
--- PASS: TestShardRecoveryDataLogAhead (23.16s)
=== RUN   TestShardRecoveryDestroyBlksInDataLog
--- PASS: TestShardRecoveryDestroyBlksInDataLog (10.53s)
=== RUN   TestShardRecoveryDestroyBlksInRecoveryLog
--- PASS: TestShardRecoveryDestroyBlksInRecoveryLog (11.09s)
=== RUN   TestShardRecoveryDestroyBlksInBothLog
--- PASS: TestShardRecoveryDestroyBlksInBothLog (10.31s)
=== RUN   TestShardRecoveryRecoveryLogCorruption
--- PASS: TestShardRecoveryRecoveryLogCorruption (10.31s)
=== RUN   TestShardRecoveryDataLogCorruption
--- PASS: TestShardRecoveryDataLogCorruption (11.12s)
=== RUN   TestShardRecoverySharedNoRP
--- PASS: TestShardRecoverySharedNoRP (11.16s)
=== RUN   TestShardRecoveryNotEnoughMem
--- PASS: TestShardRecoveryNotEnoughMem (25.52s)
=== RUN   TestShardRecoveryCleanup
--- PASS: TestShardRecoveryCleanup (0.54s)
=== RUN   TestShardRecoveryRebuildSharedLog
--- PASS: TestShardRecoveryRebuildSharedLog (1.28s)
=== RUN   TestShardRecoveryUpgradeWithCheckpoint
--- PASS: TestShardRecoveryUpgradeWithCheckpoint (0.54s)
=== RUN   TestShardRecoveryUpgradeWithLogReplay
--- PASS: TestShardRecoveryUpgradeWithLogReplay (0.51s)
=== RUN   TestShardRecoveryRebuildAfterError
--- PASS: TestShardRecoveryRebuildAfterError (1.27s)
=== RUN   TestShardRecoveryRebuildAfterConcurrentDelete
--- PASS: TestShardRecoveryRebuildAfterConcurrentDelete (1.84s)
=== RUN   TestShardRecoveryAfterDeleteInstance
--- PASS: TestShardRecoveryAfterDeleteInstance (0.20s)
=== RUN   TestShardRecoveryDestroyShard
--- PASS: TestShardRecoveryDestroyShard (0.30s)
=== RUN   TestHeaderRepair
--- PASS: TestHeaderRepair (0.15s)
=== RUN   TestCheckpointWithWriter
--- PASS: TestCheckpointWithWriter (3.72s)
=== RUN   TestPlasmaRecoveryWithRepairFullReplay
--- PASS: TestPlasmaRecoveryWithRepairFullReplay (36.91s)
=== RUN   TestPlasmaRecoveryWithInsertRepairCheckpoint
--- PASS: TestPlasmaRecoveryWithInsertRepairCheckpoint (37.94s)
=== RUN   TestPlasmaRecoveryWithDeleteRepairCheckpoint
--- PASS: TestPlasmaRecoveryWithDeleteRepairCheckpoint (13.75s)
=== RUN   TestShardRecoverySharedFullReplayOnError
--- PASS: TestShardRecoverySharedFullReplayOnError (12.77s)
=== RUN   TestShardRecoveryDedicatedFullReplayOnError
--- PASS: TestShardRecoveryDedicatedFullReplayOnError (12.86s)
=== RUN   TestShardRecoverySharedFullReplayOnErrorWithRepair
--- PASS: TestShardRecoverySharedFullReplayOnErrorWithRepair (15.28s)
=== RUN   TestGlobalWorkContextForRecovery
--- PASS: TestGlobalWorkContextForRecovery (0.40s)
=== RUN   TestSkipLogSimple
--- PASS: TestSkipLogSimple (0.00s)
=== RUN   TestSkipLogLoadStore
--- PASS: TestSkipLogLoadStore (0.00s)
=== RUN   TestShardMetadata
--- FAIL: TestShardMetadata (0.00s)
github.com/couchbase/plasma.(*LSSCtx).isSharedLog(...)
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:492
github.com/couchbase/plasma.(*Shard).AddInstance(0x81a980?, 0xc015000000, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:1080 +0x132 fp=0xc0001bf028 sp=0xc0001be7e0 pc=0x644272
github.com/couchbase/plasma.testShardMetadata(_, {{0x0, 0x0}, 0xc8, 0x190, 0x19, 0x0, 0x81b3a0, 0x81b3c8, 0x81b3b0, ...})
	/opt/build/goproj/src/github.com/couchbase/plasma/shard_test.go:51 +0x353 fp=0xc0001bf688 sp=0xc0001bf028 pc=0x70f9f3
github.com/couchbase/plasma.runTest(0xc00c373ba0, {0x7f67a2, 0x11}, 0x81c008, {0x7f28e2, 0x7}, 0x0, 0x0)
	/opt/build/goproj/src/github.com/couchbase/plasma/testing.go:434 +0x3e2 fp=0xc0001bff28 sp=0xc0001bf688 pc=0x663002
github.com/couchbase/plasma.TestShardMetadata(0xc00c373a00?)
	/opt/build/goproj/src/github.com/couchbase/plasma/shard_test.go:1930 +0x3f fp=0xc0001bff70 sp=0xc0001bff28 pc=0x71dc7f
github.com/couchbase/plasma.(*smrManager).run(0xc000124f60)
	/opt/build/goproj/src/github.com/couchbase/plasma/smr.go:520 +0x96 fp=0xc0000537c8 sp=0xc000053768 pc=0x651456
github.com/couchbase/plasma.NewSmrManager.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/smr.go:428 +0x26 fp=0xc0000537e0 sp=0xc0000537c8 pc=0x6510a6
created by github.com/couchbase/plasma.NewSmrManager
	/opt/build/goproj/src/github.com/couchbase/plasma/smr.go:428 +0xaf
github.com/couchbase/plasma.runCleanerAutoTuner()
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner.go:653 +0x194 fp=0xc000053fe0 sp=0xc000053f68 pc=0x5ddfd4
created by github.com/couchbase/plasma.init.2
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:198 +0x149
github.com/couchbase/plasma.singletonWorker()
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2452 +0xb0 fp=0xc0000547e0 sp=0xc000054780 pc=0x64d0d0
created by github.com/couchbase/plasma.init.2
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:199 +0x155
github.com/couchbase/plasma.systemResourceTracker()
	/opt/build/goproj/src/github.com/couchbase/plasma/mem.go:533 +0xa5 fp=0xc000054fe0 sp=0xc000054f60 pc=0x738c25
created by github.com/couchbase/plasma.init.2
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:200 +0x165
github.com/couchbase/plasma.AggregateAndLogStats()
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2417 +0x147 fp=0xc000069fe0 sp=0xc000069c98 pc=0x64b827
created by github.com/couchbase/plasma.init.2
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:201 +0x171
github.com/couchbase/plasma.TestExtrasN1(0x100000000650c19?)
	/opt/build/goproj/src/github.com/couchbase/plasma/extras_test.go:25 +0x2b fp=0xc00006bf70 sp=0xc00006bed0 pc=0x67daeb
github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0xc00028c100)
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner.go:664 +0x86 fp=0xc00029c7c8 sp=0xc00029c768 pc=0x5de166
github.com/couchbase/plasma.runCleanerAutoTuner.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner.go:650 +0x26 fp=0xc00029c7e0 sp=0xc00029c7c8 pc=0x5de046
created by github.com/couchbase/plasma.runCleanerAutoTuner
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner.go:650 +0x136
github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc0000f2000)
	/opt/build/goproj/src/github.com/couchbase/plasma/mvcc_purger.go:132 +0x2c5 fp=0xc00009ffc8 sp=0xc00009f1d8 pc=0x6071c5
github.com/couchbase/plasma.(*Plasma).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x26 fp=0xc00009ffe0 sp=0xc00009ffc8 pc=0x61ebc6
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x1c5
github.com/couchbase/plasma.(*LSSCtx).runtimeStats(0xc0002b0500, 0xc000095560)
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:699 +0x325 fp=0xc0001cefc0 sp=0xc0001cee88 pc=0x5ffe45
github.com/couchbase/plasma.(*LSSCtx).startDaemons.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0x2a fp=0xc0001cefe0 sp=0xc0001cefc0 pc=0x5fedca
created by github.com/couchbase/plasma.(*LSSCtx).startDaemons
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0xde
github.com/couchbase/plasma.testAutoTuneFlushBufferGetUsedMemory.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner_test.go:1208 +0x75 fp=0xc000271fe0 sp=0xc000270590 pc=0x674035
created by github.com/couchbase/plasma.testAutoTuneFlushBufferGetUsedMemory
	/opt/build/goproj/src/github.com/couchbase/plasma/auto_tuner_test.go:1205 +0x415
github.com/couchbase/plasma.(*Plasma).monitor(0xc0000f2000)
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:538 +0x34a fp=0xc0001c3fc8 sp=0xc0001c27d0 pc=0x61efca
github.com/couchbase/plasma.(*Plasma).start.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x26 fp=0xc0001c3fe0 sp=0xc0001c3fc8 pc=0x61eb66
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x256
github.com/couchbase/plasma.(*Plasma).monitor(0xc000131100)
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:538 +0x34a fp=0xc0000d9fc8 sp=0xc0000d87d0 pc=0x61efca
github.com/couchbase/plasma.(*Plasma).start.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x26 fp=0xc0000d9fe0 sp=0xc0000d9fc8 pc=0x61eb66
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x256
github.com/couchbase/plasma.(*Plasma).monitor(0xc0000f2700)
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:538 +0x34a fp=0xc0000e1fc8 sp=0xc0000e07d0 pc=0x61efca
github.com/couchbase/plasma.(*Plasma).start.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x26 fp=0xc0000e1fe0 sp=0xc0000e1fc8 pc=0x61eb66
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x256
github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc000131100)
	/opt/build/goproj/src/github.com/couchbase/plasma/mvcc_purger.go:132 +0x2c5 fp=0xc000161fc8 sp=0xc0001611d8 pc=0x6071c5
github.com/couchbase/plasma.(*Plasma).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x26 fp=0xc000161fe0 sp=0xc000161fc8 pc=0x61ebc6
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x1c5
github.com/couchbase/plasma.TestExtrasN2(0x100000000650c19?)
	/opt/build/goproj/src/github.com/couchbase/plasma/extras_test.go:34 +0x19 fp=0xc000229f70 sp=0xc000229f40 pc=0x67ddd9
github.com/couchbase/plasma.(*LSSCtx).runtimeStats(0xc0002b0a00, 0xc000095e60)
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:699 +0x325 fp=0xc0001cffc0 sp=0xc0001cfe88 pc=0x5ffe45
github.com/couchbase/plasma.(*LSSCtx).startDaemons.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0x2a fp=0xc0001cffe0 sp=0xc0001cffc0 pc=0x5fedca
created by github.com/couchbase/plasma.(*LSSCtx).startDaemons
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0xde
github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc000131800)
	/opt/build/goproj/src/github.com/couchbase/plasma/mvcc_purger.go:132 +0x2c5 fp=0xc0000a1fc8 sp=0xc0000a11d8 pc=0x6071c5
github.com/couchbase/plasma.(*Plasma).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x26 fp=0xc0000a1fe0 sp=0xc0000a1fc8 pc=0x61ebc6
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x1c5
github.com/couchbase/plasma.(*Timer).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2819 +0xee fp=0xc000226fe0 sp=0xc000226f38 pc=0x64e98e
created by github.com/couchbase/plasma.(*Timer).start
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2799 +0x65
github.com/couchbase/plasma.(*LSSCtx).runtimeStats(0xc0002b0640, 0xc0000956e0)
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:699 +0x325 fp=0xc0001d07c0 sp=0xc0001d0688 pc=0x5ffe45
github.com/couchbase/plasma.(*LSSCtx).startDaemons.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0x2a fp=0xc0001d07e0 sp=0xc0001d07c0 pc=0x5fedca
created by github.com/couchbase/plasma.(*LSSCtx).startDaemons
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0xde
github.com/couchbase/plasma.(*LSSCtx).runtimeStats(0xc0002b0b40, 0xc000095ec0)
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:699 +0x325 fp=0xc0002267c0 sp=0xc000226688 pc=0x5ffe45
github.com/couchbase/plasma.(*LSSCtx).startDaemons.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0x2a fp=0xc0002267e0 sp=0xc0002267c0 pc=0x5fedca
created by github.com/couchbase/plasma.(*LSSCtx).startDaemons
	/opt/build/goproj/src/github.com/couchbase/plasma/lssctx.go:484 +0xde
github.com/couchbase/plasma.(*Timer).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2819 +0xee fp=0xc0001d4fe0 sp=0xc0001d4f38 pc=0x64e98e
created by github.com/couchbase/plasma.(*Timer).start
	/opt/build/goproj/src/github.com/couchbase/plasma/shard.go:2799 +0x65
github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc0000f2700)
	/opt/build/goproj/src/github.com/couchbase/plasma/mvcc_purger.go:132 +0x2c5 fp=0xc0000a3fc8 sp=0xc0000a31d8 pc=0x6071c5
github.com/couchbase/plasma.(*Plasma).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x26 fp=0xc0000a3fe0 sp=0xc0000a3fc8 pc=0x61ebc6
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x1c5
github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc00012e700)
	/opt/build/goproj/src/github.com/couchbase/plasma/mvcc_purger.go:132 +0x2c5 fp=0xc00015ffc8 sp=0xc00015f1d8 pc=0x6071c5
github.com/couchbase/plasma.(*Plasma).start.func1()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x26 fp=0xc00015ffe0 sp=0xc00015ffc8 pc=0x61ebc6
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:492 +0x1c5
github.com/couchbase/plasma.(*Plasma).monitor(0xc00012e700)
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:538 +0x34a fp=0xc0000ddfc8 sp=0xc0000dc7d0 pc=0x61efca
github.com/couchbase/plasma.(*Plasma).start.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x26 fp=0xc0000ddfe0 sp=0xc0000ddfc8 pc=0x61eb66
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x256
github.com/couchbase/plasma.(*Plasma).monitor(0xc000131800)
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:538 +0x34a fp=0xc0000e5fc8 sp=0xc0000e47d0 pc=0x61efca
github.com/couchbase/plasma.(*Plasma).start.func2()
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x26 fp=0xc0000e5fe0 sp=0xc0000e5fc8 pc=0x61eb66
created by github.com/couchbase/plasma.(*Plasma).start
	/opt/build/goproj/src/github.com/couchbase/plasma/plasma.go:501 +0x256
github.com/couchbase/plasma.TestExtrasN3(0x100000000650c19?)
	/opt/build/goproj/src/github.com/couchbase/plasma/extras_test.go:40 +0x19 fp=0xc00022c770 sp=0xc00022c740 pc=0x67de39
FAIL	github.com/couchbase/plasma	3675.330s
=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.034s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-07-01T13:23:49.224+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5760 us

2022-07-01T13:23:49.225+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-01T13:23:49.229+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-07-01T13:23:49.229+05:30 [INFO][FDB] Forestdb opened database file test
2022-07-01T13:23:49.232+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 56.644448374s -> 176539.80729010038 items/s
Incr build: 10000000 items took 2m53.220878396s -> 57729.761519503525 items/s
Main Index: {
"node_count":             18000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       4,
"next_pointers_per_node": 1.3333,
"memory_used":            1695887724,
"node_allocs":            18000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 13500190,
"level1": 3375549,
"level2": 843074,
"level3": 210794,
"level4": 52734,
"level5": 13349,
"level6": 3244,
"level7": 782,
"level8": 221,
"level9": 42,
"level10": 17,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (229.87s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	253.976s
=== RUN   TestConnPoolBasicSanity
2022-07-01T13:28:06.444+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-07-01T13:28:06.652+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:28:07.445+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-07-01T13:28:11.448+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-07-01T13:28:51.204+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:28:51.465+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.76s)
=== RUN   TestLongevity
2022-07-01T13:28:55.206+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-07-01T13:29:33.600+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:29:34.226+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.40s)
=== RUN   TestSustainedHighConns
2022-07-01T13:29:37.602+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Returning 1 Connections
cp.curActConns = 0
Allocating 8 Connections
cp.curActConns = 7
Returning 2 Connections
Allocating 6 Connections
cp.curActConns = 11
Returning 4 Connections
Allocating 6 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 0 Connections
cp.curActConns = 17
Returning 1 Connections
Allocating 13 Connections
cp.curActConns = 22
Returning 0 Connections
cp.curActConns = 29
Returning 4 Connections
Allocating 13 Connections
cp.curActConns = 34
Returning 3 Connections
cp.curActConns = 35
Allocating 22 Connections
Returning 3 Connections
cp.curActConns = 45
Returning 4 Connections
cp.curActConns = 50
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 49
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 56
Returning 4 Connections
cp.curActConns = 64
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 60
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 67
Returning 3 Connections
cp.curActConns = 70
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 72
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 78
Returning 0 Connections
cp.curActConns = 81
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 90
Returning 3 Connections
cp.curActConns = 93
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 94
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 101
Returning 3 Connections
cp.curActConns = 109
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 110
Returning 3 Connections
Allocating 22 Connections
cp.curActConns = 113
Returning 4 Connections
cp.curActConns = 124
Returning 2 Connections
cp.curActConns = 123
Allocating 13 Connections
Returning 1 Connections
cp.curActConns = 135
Returning 0 Connections
Allocating 23 Connections
cp.curActConns = 137
Returning 3 Connections
cp.curActConns = 149
Returning 2 Connections
cp.curActConns = 153
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 161
Returning 4 Connections
cp.curActConns = 162
Allocating 8 Connections
Returning 3 Connections
cp.curActConns = 167
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 168
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 172
Returning 1 Connections
cp.curActConns = 175
Returning 2 Connections
Allocating 15 Connections
cp.curActConns = 184
Returning 3 Connections
cp.curActConns = 185
Allocating 2 Connections
Returning 3 Connections
cp.curActConns = 184
Allocating 5 Connections
Returning 3 Connections
cp.curActConns = 186
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 187
Returning 0 Connections
Allocating 7 Connections
cp.curActConns = 194
Returning 2 Connections
Allocating 10 Connections
cp.curActConns = 195
Returning 0 Connections
cp.curActConns = 202
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 202
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 203
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 201
Returning 2 Connections
Allocating 9 Connections
cp.curActConns = 207
Returning 2 Connections
cp.curActConns = 206
Allocating 2 Connections
Returning 2 Connections
cp.curActConns = 206
Allocating 1 Connections
Returning 4 Connections
cp.curActConns = 203
Allocating 1 Connections
Returning 2 Connections
cp.curActConns = 202
Allocating 1 Connections
Returning 4 Connections
cp.curActConns = 199
Allocating 16 Connections
Returning 0 Connections
cp.curActConns = 215
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 213
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 208
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 208
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 210
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 207
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 207
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 210
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 212
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 212
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 214
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 215
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 217
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 217
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 215
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 218
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 219
Returning 0 Connections
cp.curActConns = 222
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 219
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 217
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 217
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 219
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 222
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 221
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 220
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 223
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 222
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 221
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 225
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 225
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-07-01T13:30:32.667+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:30:33.636+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.07s)
=== RUN   TestLowWM
2022-07-01T13:30:36.669+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-07-01T13:31:36.683+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-01T13:32:36.698+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-07-01T13:32:42.181+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:32:42.700+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN   TestTotalConns
2022-07-01T13:32:46.182+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-07-01T13:33:00.357+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:33:01.190+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.18s)
=== RUN   TestUpdateTickRate
2022-07-01T13:33:04.359+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-07-01T13:33:25.208+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-07-01T13:33:25.370+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	322.816s
Starting server: attempt 1

Functional tests

# github.com/couchbase/indexing/secondary/tests/framework/secondaryindex
../framework/secondaryindex/n1qlclient.go:715:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:719:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:723:47: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:727:47: undefined: tenant.Unit
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests [build failed]
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
# github.com/couchbase/indexing/secondary/tests/framework/secondaryindex
../framework/secondaryindex/n1qlclient.go:715:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:719:48: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:723:47: undefined: tenant.Unit
../framework/secondaryindex/n1qlclient.go:727:47: undefined: tenant.Unit
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests [build failed]
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.16.6
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.16.6
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> chronicle (compile)
[100%] Built target chronicle
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target golang_tls_test using Go 1.17.4
[100%] Built target golang_tls_test
[100%] Building Go target ns_goport using Go 1.18.1
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.1
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.1
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.1
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.1
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-07-01 13:36:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:17 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:17 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 13:36:20 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 13:36:26 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:38 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:39 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-01 13:36:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:43 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-07-01 13:36:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '21', 'memoryTotal': 15466930176, 'memoryFree': 6266048512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:36:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:36:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 13632630784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:36:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:36:45 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 5976080384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:36:45 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:36:46 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 5871517696, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:36:46 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:36:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:36:46 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-01 13:36:46 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 13:36:46 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-07-01 13:36:46 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:36:51 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-01 13:36:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-01 13:36:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.04 seconds to create bucket default
2022-07-01 13:36:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-01 13:36:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:36:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:36:53 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-01 13:36:53 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-07-01 13:36:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:36:54 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 0, 'mem_free': 13522358272, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 40.431761741638184
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-01 13:36:57 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 13:37:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-01 13:37:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:03 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-07-01 13:37:03 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:37:05 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-07-01 13:37:05 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:05 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-01 13:37:05 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 99.000317ms
2022-07-01 13:37:35 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '935b140f-ba43-4c54-bc83-aa6ee657e336', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '99.000317ms', 'executionTime': '98.885651ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-01 13:37:35 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 441.934396ms
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 17.756869ms
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.679176ms
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.261666ms
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.565041313057166, 'mem_free': 13566095360, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-07-01 13:37:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 13:37:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 13:37:43 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 86.219s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1432-rel-enterprise
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.28730000706058, 'mem_free': 13517553664, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:45 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 13:37:47 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-07-01 13:37:47 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '89', 'memoryTotal': 15466930176, 'memoryFree': 13537718272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-07-01 13:37:48 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '87', 'memoryTotal': 15466930176, 'memoryFree': 13541421056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '83', 'memoryTotal': 15466930176, 'memoryFree': 13542387712, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '82', 'memoryTotal': 15466930176, 'memoryFree': 13539504128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-07-01 13:37:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-07-01 13:37:49 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-07-01 13:37:50 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-07-01 13:37:55 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-07-01 13:37:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-07-01 13:37:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.03 seconds to create bucket default
2022-07-01 13:37:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-07-01 13:37:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:37:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:37:57 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:37:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:37:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.82386282097144, 'mem_free': 13536264192, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 18.3930561542511
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-07-01 13:38:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-07-01 13:38:06 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-07-01 13:38:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:07 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-07-01 13:38:07 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-07-01 13:38:09 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-07-01 13:38:09 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:09 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-07-01 13:38:09 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.597663ms
2022-07-01 13:38:39 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'aaacbd38-5724-4887-a545-3bba8ac06760', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '21.597663ms', 'executionTime': '21.519291ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-07-01 13:38:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 426.847001ms
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.036481ms
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-07-01 13:38:40 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:41 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee3f5145ecf8134561981900611570684cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-07-01 13:38:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee3f5145ecf8134561981900611570684cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-07-01 13:38:41 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 80.251287ms
2022-07-01 13:38:41 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee3f5145ecf8134561981900611570684cjob_title) USING GSI
2022-07-01 13:38:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee3f5145ecf8134561981900611570684cjob_title) USING GSI
2022-07-01 13:38:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee3f5145ecf8134561981900611570684cjob_title%29+USING+GSI
2022-07-01 13:38:42 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 18.799125ms
2022-07-01 13:38:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee3f5145ecf8134561981900611570684cjob_title'
2022-07-01 13:38:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f5145ecf8134561981900611570684cjob_title%27
2022-07-01 13:38:43 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.770752ms
2022-07-01 13:38:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee3f5145ecf8134561981900611570684cjob_title'
2022-07-01 13:38:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f5145ecf8134561981900611570684cjob_title%27
2022-07-01 13:38:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.744726ms
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.698878ms
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'a37f92c9-2328-4128-acf2-e0451716b99a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3f5145ecf8134561981900611570684cjob_title', 'index_id': 'a2dea802d14170d3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.698878ms', 'executionTime': '3.628793ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:45 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-07-01 13:38:45 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 78.081853ms
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-07-01 13:38:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-07-01 13:38:47 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:47 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee3f5145ecf8134561981900611570684cjob_title'
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f5145ecf8134561981900611570684cjob_title%27
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.474903ms
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee3f5145ecf8134561981900611570684cjob_title ON default USING GSI
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee3f5145ecf8134561981900611570684cjob_title+ON+default+USING+GSI
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 40.927993ms
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee3f5145ecf8134561981900611570684cjob_title'
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3f5145ecf8134561981900611570684cjob_title%27
2022-07-01 13:38:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 1.359612ms
2022-07-01 13:38:48 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '9361a796-8afd-4079-ae22-bbbb92860822', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.359612ms', 'executionTime': '1.295111ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:48 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.696153ms
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.309771ms
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.96970681872844, 'mem_free': 13398581248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-07-01 13:38:50 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-07-01 13:38:50 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-07-01 13:38:54 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 13:38:54 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-07-01 13:38:55 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-07-01 13:38:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 72.385s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_2'}
[2022-07-01 13:38:55,496] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:38:55,496] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:38:55,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:38:55,801] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:38:55,810] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:38:55,810] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-07-01 13:38:55,811] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:38:56,500] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:38:56,504] - [task:161] INFO -  {'uptime': '156', 'memoryTotal': 15466930176, 'memoryFree': 13398581248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:38:56,506] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:38:56,507] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:38:56,507] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:38:56,514] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:38:56,519] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:38:56,520] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:38:56,523] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:38:56,524] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:38:56,524] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:38:56,525] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:38:56,527] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:38:56,530] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:38:56,530] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:38:56,791] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:38:56,792] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:38:56,836] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:38:56,837] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:38:56,842] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:56,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:56,850] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:38:56,874] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:38:56,878] - [task:161] INFO -  {'uptime': '152', 'memoryTotal': 15466930176, 'memoryFree': 13452230656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:38:56,880] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:38:56,886] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:38:56,886] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:38:56,889] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:38:56,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:38:56,892] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:38:57,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:38:57,163] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:38:57,206] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:38:57,208] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:38:57,212] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,214] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,219] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:38:57,238] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:38:57,241] - [task:161] INFO -  {'uptime': '153', 'memoryTotal': 15466930176, 'memoryFree': 13452517376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:38:57,243] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:38:57,247] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:38:57,247] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:38:57,251] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:38:57,257] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:38:57,257] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:38:57,518] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:38:57,519] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:38:57,563] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:38:57,564] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:38:57,569] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,571] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,577] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:38:57,596] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:38:57,599] - [task:161] INFO -  {'uptime': '148', 'memoryTotal': 15466930176, 'memoryFree': 13406511104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:38:57,601] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:38:57,606] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:38:57,606] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:38:57,608] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:38:57,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:38:57,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:38:57,872] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:38:57,873] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:38:57,916] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:38:57,917] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:38:57,922] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,924] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:38:57,930] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:38:57,948] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:38:58,065] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:39:03,067] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:39:03,080] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:39:03,085] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:03,086] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:03,364] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:03,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:39:03,409] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:39:03,410] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:39:03,410] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:39:03,967] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:39:03,997] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:39:03,998] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:39:04,558] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:39:04,599] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:39:05,252] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:39:05,254] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:39:05,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:05,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:05,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:05,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:05,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:05,815] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:05,820] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:05,820] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:06,068] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:06,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:06,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:06,411] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:10,073] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:39:10,074] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.4598657481611, 'mem_free': 13449289728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:39:10,074] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:39:10,074] - [basetestcase:467] INFO - Time to execute basesetup : 14.58080005645752
[2022-07-01 13:39:10,078] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:39:10,079] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:39:10,085] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:39:10,086] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:39:10,091] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:39:10,091] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:39:10,098] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:39:10,098] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:39:10,102] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:10,113] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:39:10,114] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:39:10,114] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:39:15,122] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:39:15,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:15,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:15,383] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:16,288] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:39:16,385] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:39:18,447] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:39:18,465] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 13:39:19,096] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:19,096] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:39:19,096] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:39:49,119] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:39:49,124] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:39:49,127] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:39:49,148] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.341615ms
[2022-07-01 13:39:49,148] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '10e4ce4b-eb65-4f69-8c57-5254de86d0d3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.341615ms', 'executionTime': '19.268833ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:39:49,148] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:39:49,151] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:39:49,153] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:39:49,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 433.243585ms
[2022-07-01 13:39:49,588] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:39:49,596] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:39:49,603] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:39:49,622] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 15.570909ms
[2022-07-01 13:39:49,654] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:49,665] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:39:49,681] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:39:49,682] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:39:49,694] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:39:49,706] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:50,648] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeefabcb62c6c840a6854d5da04ada4fcdjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:39:50,650] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:39:50,711] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.904416ms
[2022-07-01 13:39:50,712] - [base_gsi:282] INFO - BUILD INDEX on default(employeeefabcb62c6c840a6854d5da04ada4fcdjob_title) USING GSI
[2022-07-01 13:39:51,717] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeefabcb62c6c840a6854d5da04ada4fcdjob_title) USING GSI
[2022-07-01 13:39:51,720] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%29+USING+GSI
[2022-07-01 13:39:51,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.047434ms
[2022-07-01 13:39:52,748] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeefabcb62c6c840a6854d5da04ada4fcdjob_title'
[2022-07-01 13:39:52,751] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%27
[2022-07-01 13:39:52,760] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.023618ms
[2022-07-01 13:39:53,766] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeefabcb62c6c840a6854d5da04ada4fcdjob_title'
[2022-07-01 13:39:53,768] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%27
[2022-07-01 13:39:53,777] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.212576ms
[2022-07-01 13:39:54,177] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-01 13:39:54,301] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:39:55,177] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:39:55,780] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:55,787] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:39:55,790] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 13:39:55,795] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.074917ms
[2022-07-01 13:39:55,795] - [task:3216] INFO - {'requestID': '390c49ee-fc4f-4dfa-b318-3f62b7f7d031', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeefabcb62c6c840a6854d5da04ada4fcdjob_title', 'index_id': '8b2cfe7bbba1fb1f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.074917ms', 'executionTime': '2.986187ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 13:39:55,795] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:55,796] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:55,796] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:39:55,796] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:39:55,797] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 13:39:55,797] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:39:55,797] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:39:55,797] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:39:55,797] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:39:56,797] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:56,802] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:39:56,804] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 13:39:56,872] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.99631ms
[2022-07-01 13:39:56,873] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:39:56,873] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:39:57,640] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:57,640] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:39:58,647] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeefabcb62c6c840a6854d5da04ada4fcdjob_title'
[2022-07-01 13:39:58,649] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%27
[2022-07-01 13:39:58,658] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.595697ms
[2022-07-01 13:39:58,660] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeefabcb62c6c840a6854d5da04ada4fcdjob_title ON default USING GSI
[2022-07-01 13:39:58,662] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeefabcb62c6c840a6854d5da04ada4fcdjob_title+ON+default+USING+GSI
[2022-07-01 13:39:58,708] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.023552ms
[2022-07-01 13:39:58,713] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeefabcb62c6c840a6854d5da04ada4fcdjob_title'
[2022-07-01 13:39:58,716] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeefabcb62c6c840a6854d5da04ada4fcdjob_title%27
[2022-07-01 13:39:58,719] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.859848ms
[2022-07-01 13:39:58,720] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '224e5e8e-c802-49e5-85d0-e7f9d26a39eb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.859848ms', 'executionTime': '1.787013ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:39:58,729] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:58,732] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:58,732] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:59,166] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:59,174] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:39:59,174] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:39:59,237] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:39:59,244] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:59,247] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:59,247] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:39:59,603] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:39:59,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:39:59,612] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:39:59,676] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:39:59,683] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:39:59,684] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:39:59,684] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:39:59,686] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:39:59,689] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:39:59,698] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.798656ms
[2022-07-01 13:39:59,700] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:39:59,702] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:39:59,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.728641ms
[2022-07-01 13:39:59,773] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:39:59,773] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.307189483008165, 'mem_free': 13363109888, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:39:59,774] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:39:59,780] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:39:59,780] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:00,156] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:00,160] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:00,161] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:00,764] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:00,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:00,771] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:01,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:01,422] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:01,423] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:02,034] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:05,683] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:40:05,706] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:40:06,449] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:40:06,452] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:40:06,452] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:40:06,458] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:40:06,465] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:40:06,471] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:40:06,472] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:40:06,481] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:40:06,482] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:40:06,484] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,496] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:40:06,496] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,500] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:40:06,503] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:40:06,503] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,506] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:40:06,508] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:40:06,508] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,511] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:40:06,513] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:40:06,514] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,516] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:40:06,517] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:40:06,517] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:40:06,517] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 71.028s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_3'}
[2022-07-01 13:40:06,567] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:06,567] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:06,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:06,893] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:40:06,901] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:40:06,902] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-07-01 13:40:06,902] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:40:07,572] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:40:07,575] - [task:161] INFO -  {'uptime': '227', 'memoryTotal': 15466930176, 'memoryFree': 13336244224, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:40:07,577] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:40:07,577] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:40:07,578] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:40:07,588] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:40:07,598] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:40:07,598] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:40:07,601] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:40:07,601] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:40:07,605] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:40:07,605] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:40:07,607] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:40:07,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:07,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:07,920] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:07,921] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:40:07,970] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:40:07,971] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:40:07,976] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:07,978] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:07,984] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:40:08,012] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:40:08,015] - [task:161] INFO -  {'uptime': '222', 'memoryTotal': 15466930176, 'memoryFree': 13390868480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:40:08,018] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:40:08,022] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:40:08,022] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:40:08,025] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:40:08,028] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:08,029] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:08,349] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:08,350] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:40:08,399] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:40:08,401] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:40:08,406] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:08,409] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:08,413] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:40:08,436] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:40:08,439] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13391577088, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:40:08,441] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:40:08,446] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:40:08,446] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:40:08,449] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:40:08,454] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:08,455] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:08,760] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:08,761] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:40:08,811] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:40:08,812] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:40:08,817] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:08,820] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:08,825] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:40:08,844] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:40:08,848] - [task:161] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13390921728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:40:08,851] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:40:08,855] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:40:08,856] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:40:08,859] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:40:08,864] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:08,864] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:09,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:09,196] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:40:09,245] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:40:09,247] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:40:09,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:09,254] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:40:09,260] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:40:09,281] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:40:09,409] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:40:14,412] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:40:14,423] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:40:14,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:14,428] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:14,743] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:14,744] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:40:14,793] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:40:14,794] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:40:14,794] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:40:15,299] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:40:15,330] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:40:15,331] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:40:15,891] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:40:15,931] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:40:16,589] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:40:16,644] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:40:16,654] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:16,654] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:17,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:17,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:17,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:17,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:17,324] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:17,324] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:17,663] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:17,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:17,670] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:18,234] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:22,407] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:40:22,408] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.62489839990537, 'mem_free': 13388439552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:40:22,408] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:40:22,408] - [basetestcase:467] INFO - Time to execute basesetup : 15.843284845352173
[2022-07-01 13:40:22,412] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:40:22,413] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:40:22,420] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:40:22,420] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:40:22,426] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:40:22,426] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:40:22,433] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:40:22,433] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:40:22,437] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:40:22,452] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:40:22,452] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:40:22,454] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:40:27,465] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:40:27,469] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:40:27,469] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:40:27,778] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:40:28,698] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:40:28,721] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:40:30,854] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:40:30,874] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 13:40:31,657] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:40:31,657] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:40:31,657] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:41:01,681] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:41:01,686] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:41:01,689] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:41:01,709] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.425266ms
[2022-07-01 13:41:01,709] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '75fc4513-2d71-40b8-bd11-08fa0ac8e4e4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.425266ms', 'executionTime': '18.355436ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:41:01,709] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:41:01,712] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:41:01,714] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:41:02,117] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 401.205572ms
[2022-07-01 13:41:02,117] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:41:02,125] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:41:02,130] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:41:02,141] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.342813ms
[2022-07-01 13:41:02,163] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:02,169] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:41:02,184] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:41:02,184] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:41:02,200] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:41:02,215] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:03,167] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee48d9064b57194ad6958078dcf2c891a9job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:41:03,169] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee48d9064b57194ad6958078dcf2c891a9job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:41:03,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.549414ms
[2022-07-01 13:41:03,241] - [base_gsi:282] INFO - BUILD INDEX on default(employee48d9064b57194ad6958078dcf2c891a9job_title) USING GSI
[2022-07-01 13:41:04,247] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee48d9064b57194ad6958078dcf2c891a9job_title) USING GSI
[2022-07-01 13:41:04,250] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee48d9064b57194ad6958078dcf2c891a9job_title%29+USING+GSI
[2022-07-01 13:41:04,286] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.165722ms
[2022-07-01 13:41:05,293] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d9064b57194ad6958078dcf2c891a9job_title'
[2022-07-01 13:41:05,295] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d9064b57194ad6958078dcf2c891a9job_title%27
[2022-07-01 13:41:05,305] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.672697ms
[2022-07-01 13:41:06,311] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d9064b57194ad6958078dcf2c891a9job_title'
[2022-07-01 13:41:06,313] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d9064b57194ad6958078dcf2c891a9job_title%27
[2022-07-01 13:41:06,320] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.17846ms
[2022-07-01 13:41:06,808] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 13:41:06,831] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:41:07,583] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:41:08,322] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:08,330] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:41:08,333] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 13:41:08,336] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.230589ms
[2022-07-01 13:41:08,336] - [task:3216] INFO - {'requestID': '384d2d25-cd82-4963-936c-824ca0e90227', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee48d9064b57194ad6958078dcf2c891a9job_title', 'index_id': 'c950b9c33c1ba04a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.230589ms', 'executionTime': '2.154912ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 13:41:08,337] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:08,337] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:08,337] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:41:08,338] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:41:08,338] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 13:41:08,338] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:41:08,338] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:41:08,339] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:41:08,339] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:41:09,339] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:09,344] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:41:09,346] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 13:41:09,417] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.522985ms
[2022-07-01 13:41:09,418] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:41:09,418] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:41:10,178] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:10,178] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:41:11,184] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d9064b57194ad6958078dcf2c891a9job_title'
[2022-07-01 13:41:11,187] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d9064b57194ad6958078dcf2c891a9job_title%27
[2022-07-01 13:41:11,195] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.491489ms
[2022-07-01 13:41:11,198] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee48d9064b57194ad6958078dcf2c891a9job_title ON default USING GSI
[2022-07-01 13:41:11,201] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee48d9064b57194ad6958078dcf2c891a9job_title+ON+default+USING+GSI
[2022-07-01 13:41:11,246] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.685777ms
[2022-07-01 13:41:11,258] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee48d9064b57194ad6958078dcf2c891a9job_title'
[2022-07-01 13:41:11,260] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee48d9064b57194ad6958078dcf2c891a9job_title%27
[2022-07-01 13:41:11,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.533404ms
[2022-07-01 13:41:11,264] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bc728ad7-3745-4491-b0a1-433c06c12628', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.533404ms', 'executionTime': '1.468289ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:41:11,273] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:11,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:11,278] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:11,743] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:11,750] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:41:11,751] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:41:11,820] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:41:11,827] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:11,830] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:11,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:12,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:12,280] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:41:12,281] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:41:12,353] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:41:12,361] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:12,361] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:41:12,361] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:41:12,363] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:41:12,366] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:41:12,375] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.485097ms
[2022-07-01 13:41:12,378] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:41:12,380] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:41:12,437] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.060098ms
[2022-07-01 13:41:12,448] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:41:12,449] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.19560225651677, 'mem_free': 13209161728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:41:12,449] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:41:12,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:12,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:12,868] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:12,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:12,873] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:13,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:13,288] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:13,288] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:13,696] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:13,701] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:13,701] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:14,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:19,429] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:41:19,451] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:41:20,724] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:41:20,727] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:41:20,727] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:41:20,734] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:41:20,740] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:41:20,746] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:41:20,747] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:41:20,756] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:41:20,757] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:41:20,759] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:20,770] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:41:20,770] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:20,773] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:41:20,776] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:41:20,776] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:20,779] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:41:20,784] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:41:20,785] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:20,790] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:41:20,793] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:41:20,794] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:20,797] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:41:20,798] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:41:20,798] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:41:20,798] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 74.239s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_4'}
[2022-07-01 13:41:20,853] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:20,854] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:21,235] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:21,241] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:41:21,250] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:41:21,250] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-07-01 13:41:21,251] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:41:21,858] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:41:21,862] - [task:161] INFO -  {'uptime': '301', 'memoryTotal': 15466930176, 'memoryFree': 13339799552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:41:21,864] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:41:21,864] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:41:21,864] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:41:21,870] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:41:21,880] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:41:21,881] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:41:21,885] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:41:21,886] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:41:21,886] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:41:21,886] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:41:21,889] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:41:21,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:21,893] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:22,281] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:22,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:41:22,341] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:41:22,342] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:41:22,347] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:22,350] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:22,356] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:41:22,387] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:41:22,390] - [task:161] INFO -  {'uptime': '298', 'memoryTotal': 15466930176, 'memoryFree': 13272236032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:41:22,393] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:41:22,397] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:41:22,397] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:41:22,399] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:41:22,404] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:22,404] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:22,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:22,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:41:22,858] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:41:22,860] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:41:22,865] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:22,867] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:22,873] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:41:22,895] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:41:22,898] - [task:161] INFO -  {'uptime': '298', 'memoryTotal': 15466930176, 'memoryFree': 13271707648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:41:22,901] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:41:22,905] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:41:22,905] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:41:22,908] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:41:22,914] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:22,914] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:23,301] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:23,302] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:41:23,362] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:41:23,363] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:41:23,368] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:23,370] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:23,376] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:41:23,396] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:41:23,398] - [task:161] INFO -  {'uptime': '293', 'memoryTotal': 15466930176, 'memoryFree': 13340565504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:41:23,401] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:41:23,406] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:41:23,406] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:41:23,408] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:41:23,412] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:23,412] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:23,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:23,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:41:23,856] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:41:23,857] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:41:23,862] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:23,864] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:41:23,871] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:41:23,889] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:41:24,002] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:41:29,003] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:41:29,014] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:41:29,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:29,019] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:29,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:29,475] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:41:29,543] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:41:29,544] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:41:29,544] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:41:29,905] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:41:29,936] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:41:29,937] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:41:30,598] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:41:30,638] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:41:31,236] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:41:31,238] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:41:31,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:31,250] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:31,664] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:31,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:31,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:32,041] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:32,045] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:32,046] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:32,572] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:32,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:32,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:33,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:37,827] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:41:37,827] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.68444990544306, 'mem_free': 13342162944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:41:37,827] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:41:37,828] - [basetestcase:467] INFO - Time to execute basesetup : 16.976731061935425
[2022-07-01 13:41:37,832] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:41:37,832] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:41:37,839] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:41:37,839] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:41:37,845] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:41:37,845] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:41:37,851] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:41:37,851] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:41:37,857] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:37,868] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:41:37,871] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:41:37,871] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:41:42,883] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:41:42,886] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:41:42,886] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:41:43,289] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:41:44,238] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:41:44,261] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:41:46,262] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:41:46,287] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-07-01 13:41:46,935] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:41:46,935] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:41:46,936] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:42:16,959] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:42:16,966] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:42:16,969] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:42:16,992] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.25889ms
[2022-07-01 13:42:16,992] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5ee34da7-30fd-44ca-97c3-0effa42c8d24', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.25889ms', 'executionTime': '20.16418ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:42:16,992] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:42:16,995] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:42:16,997] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:42:17,432] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 432.451086ms
[2022-07-01 13:42:17,432] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:42:17,445] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:42:17,449] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:42:17,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.355447ms
[2022-07-01 13:42:17,497] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:17,501] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:42:17,519] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:42:17,520] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:42:17,531] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:42:17,542] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:18,490] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3201754909ae43628a928fa0a6866440job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:42:18,492] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3201754909ae43628a928fa0a6866440job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:42:18,560] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.598691ms
[2022-07-01 13:42:18,560] - [base_gsi:282] INFO - BUILD INDEX on default(employee3201754909ae43628a928fa0a6866440job_title) USING GSI
[2022-07-01 13:42:19,567] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3201754909ae43628a928fa0a6866440job_title) USING GSI
[2022-07-01 13:42:19,570] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3201754909ae43628a928fa0a6866440job_title%29+USING+GSI
[2022-07-01 13:42:19,593] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.20262ms
[2022-07-01 13:42:20,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3201754909ae43628a928fa0a6866440job_title'
[2022-07-01 13:42:20,602] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3201754909ae43628a928fa0a6866440job_title%27
[2022-07-01 13:42:20,612] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.962843ms
[2022-07-01 13:42:21,618] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3201754909ae43628a928fa0a6866440job_title'
[2022-07-01 13:42:21,620] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3201754909ae43628a928fa0a6866440job_title%27
[2022-07-01 13:42:21,629] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.44251ms
[2022-07-01 13:42:22,133] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 13:42:22,157] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:42:22,938] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:42:22,989] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-07-01 13:42:22,989] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:42:23,020] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-07-01 13:42:23,020] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-07-01 13:42:23,021] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3273865191, 0, b'')
[2022-07-01 13:42:23,632] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:23,635] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:42:23,638] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 13:42:23,642] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.390346ms
[2022-07-01 13:42:23,642] - [task:3216] INFO - {'requestID': 'a7d5e8a8-4588-4ee6-8436-03cd91ffb4dd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee3201754909ae43628a928fa0a6866440job_title', 'index_id': 'e60be8f57e650266', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.390346ms', 'executionTime': '2.317758ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 13:42:23,642] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:23,643] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:23,643] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:42:23,643] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:42:23,644] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 13:42:23,644] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:42:23,644] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:42:23,645] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:42:23,645] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:42:24,644] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:24,650] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:42:24,652] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 13:42:24,735] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.576714ms
[2022-07-01 13:42:24,736] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:42:24,736] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:42:25,517] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:25,517] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:42:26,524] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3201754909ae43628a928fa0a6866440job_title'
[2022-07-01 13:42:26,526] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3201754909ae43628a928fa0a6866440job_title%27
[2022-07-01 13:42:26,535] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.788694ms
[2022-07-01 13:42:26,537] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3201754909ae43628a928fa0a6866440job_title ON default USING GSI
[2022-07-01 13:42:26,539] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3201754909ae43628a928fa0a6866440job_title+ON+default+USING+GSI
[2022-07-01 13:42:26,572] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.479752ms
[2022-07-01 13:42:26,580] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3201754909ae43628a928fa0a6866440job_title'
[2022-07-01 13:42:26,583] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3201754909ae43628a928fa0a6866440job_title%27
[2022-07-01 13:42:26,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.388166ms
[2022-07-01 13:42:26,586] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '57d6264e-286a-46f8-9a3b-5dafe2bf499b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.388166ms', 'executionTime': '1.32178ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:42:26,594] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:26,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:26,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:27,093] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:27,101] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:42:27,101] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:42:27,181] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:42:27,189] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:27,192] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:27,192] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:27,633] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:27,643] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:42:27,643] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:42:27,718] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:42:27,726] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:27,726] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:42:27,726] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:42:27,728] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:42:27,730] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:42:27,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.609482ms
[2022-07-01 13:42:27,743] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:42:27,745] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:42:27,804] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.493981ms
[2022-07-01 13:42:27,815] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:42:27,816] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.091812262282886, 'mem_free': 13234982912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:42:27,816] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:42:27,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:27,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:28,291] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:28,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:28,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:28,733] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:28,738] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:28,739] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:29,318] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:29,325] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:29,326] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:30,098] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:35,571] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:42:35,750] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:42:36,348] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:42:36,350] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:42:36,351] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:42:36,357] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:42:36,364] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:42:36,371] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:42:36,372] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:42:36,381] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:42:36,382] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:42:36,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,397] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:42:36,397] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,400] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:42:36,402] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:42:36,402] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,406] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:42:36,408] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:42:36,409] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,412] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:42:36,414] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:42:36,414] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,417] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:42:36,417] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:42:36,418] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:42:36,418] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 75.571s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_5'}
[2022-07-01 13:42:36,455] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:36,456] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:36,888] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:36,893] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:42:36,901] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:42:36,902] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-07-01 13:42:36,902] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:42:37,460] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:42:37,463] - [task:161] INFO -  {'uptime': '377', 'memoryTotal': 15466930176, 'memoryFree': 13191716864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:42:37,466] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:42:37,466] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:42:37,466] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:42:37,471] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:42:37,477] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:42:37,477] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:42:37,481] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:42:37,482] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:42:37,482] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:42:37,482] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:42:37,485] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:42:37,490] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:37,490] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:37,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:37,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:42:37,976] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:42:37,977] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:42:37,982] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:37,985] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:37,990] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:42:38,019] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:42:38,022] - [task:161] INFO -  {'uptime': '373', 'memoryTotal': 15466930176, 'memoryFree': 13298970624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:42:38,024] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:42:38,028] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:42:38,029] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:42:38,031] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:42:38,038] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:38,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:38,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:38,456] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:42:38,521] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:42:38,522] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:42:38,526] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:38,529] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:38,534] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:42:38,554] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:42:38,557] - [task:161] INFO -  {'uptime': '374', 'memoryTotal': 15466930176, 'memoryFree': 13299359744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:42:38,559] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:42:38,564] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:42:38,564] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:42:38,568] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:42:38,572] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:38,572] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:38,999] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:39,001] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:42:39,066] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:42:39,067] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:42:39,073] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:39,076] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:39,080] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:42:39,102] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:42:39,105] - [task:161] INFO -  {'uptime': '374', 'memoryTotal': 15466930176, 'memoryFree': 13298352128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:42:39,107] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:42:39,112] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:42:39,113] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:42:39,116] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:42:39,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:39,120] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:39,545] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:39,546] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:42:39,613] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:42:39,614] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:42:39,619] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:39,621] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:42:39,627] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:42:39,646] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:42:39,763] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:42:44,766] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:42:44,777] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:42:44,782] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:44,782] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:45,229] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:45,230] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:42:45,295] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:42:45,296] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:42:45,296] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:42:45,664] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:42:45,697] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:42:45,698] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:42:46,260] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:42:46,298] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:42:46,960] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:42:46,963] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:42:46,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:46,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:47,435] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:47,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:47,440] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:47,844] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:47,850] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:47,850] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:48,541] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:48,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:48,548] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:49,298] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:42:54,357] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:42:54,357] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.11744346995847, 'mem_free': 13299687424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:42:54,358] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:42:54,358] - [basetestcase:467] INFO - Time to execute basesetup : 17.904865741729736
[2022-07-01 13:42:54,363] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:42:54,363] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:42:54,369] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:42:54,369] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:42:54,376] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:42:54,376] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:42:54,382] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:42:54,382] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:42:54,386] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:42:54,399] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:42:54,399] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:42:54,400] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:42:59,412] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:42:59,414] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:42:59,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:42:59,845] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:00,900] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:43:00,925] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:43:02,784] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:43:02,802] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-01 13:43:03,821] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:03,821] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:43:03,822] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:43:33,832] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:43:33,837] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:43:33,840] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:43:33,860] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.065797ms
[2022-07-01 13:43:33,861] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ab90f08b-a6bc-4a0a-99b5-05deb71d8474', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.065797ms', 'executionTime': '18.995122ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:43:33,861] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:43:33,863] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:43:33,865] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:43:34,287] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 419.566519ms
[2022-07-01 13:43:34,287] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:43:34,294] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:43:34,302] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:43:34,314] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.157814ms
[2022-07-01 13:43:34,337] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:34,343] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:43:34,365] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:43:34,365] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:43:34,381] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:43:34,400] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:35,337] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee0ff50abc5bf846f3845060326ec0c5b6job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:43:35,339] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee0ff50abc5bf846f3845060326ec0c5b6job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:43:35,409] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.069102ms
[2022-07-01 13:43:35,410] - [base_gsi:282] INFO - BUILD INDEX on default(employee0ff50abc5bf846f3845060326ec0c5b6job_title) USING GSI
[2022-07-01 13:43:36,416] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee0ff50abc5bf846f3845060326ec0c5b6job_title) USING GSI
[2022-07-01 13:43:36,419] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee0ff50abc5bf846f3845060326ec0c5b6job_title%29+USING+GSI
[2022-07-01 13:43:36,439] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.471192ms
[2022-07-01 13:43:37,446] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0ff50abc5bf846f3845060326ec0c5b6job_title'
[2022-07-01 13:43:37,448] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0ff50abc5bf846f3845060326ec0c5b6job_title%27
[2022-07-01 13:43:37,456] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.07787ms
[2022-07-01 13:43:38,463] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0ff50abc5bf846f3845060326ec0c5b6job_title'
[2022-07-01 13:43:38,465] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0ff50abc5bf846f3845060326ec0c5b6job_title%27
[2022-07-01 13:43:38,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.084841ms
[2022-07-01 13:43:38,991] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:43:39,015] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:43:41,632] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:43:42,480] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:42,488] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:43:42,490] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 13:43:42,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.900449ms
[2022-07-01 13:43:42,495] - [task:3216] INFO - {'requestID': '2803362d-4c4f-4cf1-a9a5-c4ecfd4c2a10', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0ff50abc5bf846f3845060326ec0c5b6job_title', 'index_id': '56f5984c80c76cb1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.900449ms', 'executionTime': '2.830513ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 13:43:42,496] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:42,496] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:42,496] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:43:42,497] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:43:42,497] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 13:43:42,497] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:43:42,498] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:43:42,498] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:43:42,498] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:43:43,498] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:43,503] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:43:43,505] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 13:43:43,622] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 101.907865ms
[2022-07-01 13:43:43,622] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:43:43,623] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:43:45,163] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:45,163] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:43:46,170] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0ff50abc5bf846f3845060326ec0c5b6job_title'
[2022-07-01 13:43:46,172] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0ff50abc5bf846f3845060326ec0c5b6job_title%27
[2022-07-01 13:43:46,182] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.957321ms
[2022-07-01 13:43:46,185] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee0ff50abc5bf846f3845060326ec0c5b6job_title ON default USING GSI
[2022-07-01 13:43:46,187] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee0ff50abc5bf846f3845060326ec0c5b6job_title+ON+default+USING+GSI
[2022-07-01 13:43:46,251] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.680429ms
[2022-07-01 13:43:46,259] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee0ff50abc5bf846f3845060326ec0c5b6job_title'
[2022-07-01 13:43:46,262] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0ff50abc5bf846f3845060326ec0c5b6job_title%27
[2022-07-01 13:43:46,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.448781ms
[2022-07-01 13:43:46,266] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'aa688bb1-0c6b-47d1-9cfd-4d81429dcfb7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.448781ms', 'executionTime': '1.353902ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:43:46,279] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:46,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:46,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:46,857] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:46,865] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:43:46,866] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:43:46,952] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:43:46,960] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:46,963] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:46,963] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:47,526] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:47,538] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:43:47,538] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:43:47,623] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:43:47,631] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:43:47,631] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:43:47,632] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:43:47,634] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:43:47,636] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:43:47,646] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.068365ms
[2022-07-01 13:43:47,648] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:43:47,650] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:43:47,705] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.626387ms
[2022-07-01 13:43:47,721] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:43:47,722] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.27073495779134, 'mem_free': 13129289728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:43:47,722] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:43:47,729] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:47,729] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:48,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:48,258] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:48,258] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:48,973] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:48,981] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:48,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:49,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:49,921] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:49,922] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:50,844] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:56,480] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:43:56,507] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:43:57,136] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:43:57,139] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:43:57,140] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:43:57,147] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:43:57,153] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:43:57,160] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:43:57,161] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:43:57,170] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:43:57,171] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:43:57,174] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,187] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:43:57,187] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,193] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:43:57,195] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:43:57,195] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,198] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:43:57,201] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:43:57,201] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,204] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:43:57,206] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:43:57,207] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,210] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:43:57,210] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:43:57,210] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:43:57,211] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 80.762s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_6'}
[2022-07-01 13:43:57,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:57,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:57,829] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:57,836] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:43:57,844] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:43:57,845] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-07-01 13:43:57,845] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:43:58,278] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:43:58,282] - [task:161] INFO -  {'uptime': '458', 'memoryTotal': 15466930176, 'memoryFree': 13129289728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:43:58,284] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:43:58,284] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:43:58,284] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:43:58,291] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:43:58,303] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:43:58,304] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:43:58,307] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:43:58,307] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:43:58,308] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:43:58,308] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:43:58,310] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:43:58,314] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:58,314] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:58,834] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:58,835] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:43:58,915] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:43:58,916] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:43:58,922] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:43:58,924] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:43:58,930] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:43:58,960] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:43:58,964] - [task:161] INFO -  {'uptime': '453', 'memoryTotal': 15466930176, 'memoryFree': 13242433536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:43:58,967] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:43:58,973] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:43:58,973] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:43:58,975] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:43:58,978] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:58,979] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:43:59,496] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:43:59,497] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:43:59,574] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:43:59,575] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:43:59,580] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:43:59,583] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:43:59,588] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:43:59,609] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:43:59,612] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13242908672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:43:59,614] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:43:59,618] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:43:59,619] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:43:59,621] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:43:59,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:43:59,629] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:00,176] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:00,177] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:44:00,261] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:44:00,262] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:44:00,268] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:44:00,271] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:44:00,275] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:44:00,295] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:44:00,300] - [task:161] INFO -  {'uptime': '454', 'memoryTotal': 15466930176, 'memoryFree': 13242126336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:44:00,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:44:00,307] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:44:00,307] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:44:00,310] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:44:00,314] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:00,314] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:00,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:00,855] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:44:00,938] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:44:00,940] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:44:00,945] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:44:00,947] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:44:00,955] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:44:00,973] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:44:01,082] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:44:06,088] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:44:06,099] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:44:06,104] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:06,104] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:06,639] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:06,640] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:44:06,719] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:44:06,720] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:44:06,721] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:44:06,990] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:44:07,025] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:44:07,026] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:44:07,728] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:44:07,766] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:44:08,317] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:44:08,320] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:44:08,326] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:08,327] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:08,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:08,916] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:08,916] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:09,636] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:09,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:09,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:10,551] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:10,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:10,558] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:11,469] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:17,103] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:44:17,103] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.23027487157513, 'mem_free': 13244383232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:44:17,103] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:44:17,104] - [basetestcase:467] INFO - Time to execute basesetup : 19.832916021347046
[2022-07-01 13:44:17,108] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:44:17,108] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:44:17,114] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:44:17,115] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:44:17,121] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:44:17,121] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:44:17,128] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:44:17,129] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:44:17,134] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:44:17,146] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:44:17,146] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:44:17,146] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:44:22,158] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:44:22,162] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:44:22,162] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:44:22,740] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:44:23,859] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:44:23,886] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:44:26,507] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:44:26,530] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-07-01 13:44:28,037] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:44:28,037] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:44:28,038] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:44:58,066] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:44:58,072] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:44:58,074] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:44:58,094] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.247848ms
[2022-07-01 13:44:58,095] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9de2755c-0d79-45fa-ba04-2c6c30278768', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.247848ms', 'executionTime': '18.177449ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:44:58,095] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:44:58,097] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:44:58,100] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:44:58,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 394.771677ms
[2022-07-01 13:44:58,497] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:44:58,508] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:44:58,512] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:44:58,522] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.303352ms
[2022-07-01 13:44:58,560] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:44:58,563] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:44:58,584] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:44:58,585] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:44:58,598] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:44:58,613] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:44:59,552] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee1cbe881e15c40a69746d8c912c6f158job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:44:59,554] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee1cbe881e15c40a69746d8c912c6f158job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:44:59,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 75.566169ms
[2022-07-01 13:44:59,634] - [base_gsi:282] INFO - BUILD INDEX on default(employeee1cbe881e15c40a69746d8c912c6f158job_title) USING GSI
[2022-07-01 13:45:00,641] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee1cbe881e15c40a69746d8c912c6f158job_title) USING GSI
[2022-07-01 13:45:00,644] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee1cbe881e15c40a69746d8c912c6f158job_title%29+USING+GSI
[2022-07-01 13:45:00,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.334195ms
[2022-07-01 13:45:01,675] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1cbe881e15c40a69746d8c912c6f158job_title'
[2022-07-01 13:45:01,678] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1cbe881e15c40a69746d8c912c6f158job_title%27
[2022-07-01 13:45:01,684] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.287208ms
[2022-07-01 13:45:02,691] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1cbe881e15c40a69746d8c912c6f158job_title'
[2022-07-01 13:45:02,693] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1cbe881e15c40a69746d8c912c6f158job_title%27
[2022-07-01 13:45:02,699] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.829288ms
[2022-07-01 13:45:03,092] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-07-01 13:45:03,117] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:04,297] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:45:04,964] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-07-01 13:45:04,995] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:06,356] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:45:07,021] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:45:07,056] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:10,550] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:45:10,833] - [task:3206] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:10,841] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:45:10,844] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-07-01 13:45:10,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.969148ms
[2022-07-01 13:45:10,849] - [task:3216] INFO - {'requestID': '84c35fde-e535-4837-8da4-81178b2692b0', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee1cbe881e15c40a69746d8c912c6f158job_title', 'index_id': 'cf391377ae7b5bea', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.969148ms', 'executionTime': '2.892103ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-07-01 13:45:10,849] - [task:3217] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:10,850] - [task:3247] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:10,850] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:45:10,850] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:45:10,851] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-07-01 13:45:10,851] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:45:10,851] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:45:10,851] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:45:10,852] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-07-01 13:45:11,851] - [task:3206] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:11,857] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-07-01 13:45:11,859] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-07-01 13:45:11,973] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 98.083758ms
[2022-07-01 13:45:11,973] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:45:11,974] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:45:13,541] - [task:3217] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:13,542] - [task:3247] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-07-01 13:45:14,548] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1cbe881e15c40a69746d8c912c6f158job_title'
[2022-07-01 13:45:14,551] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1cbe881e15c40a69746d8c912c6f158job_title%27
[2022-07-01 13:45:14,561] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.558551ms
[2022-07-01 13:45:14,564] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee1cbe881e15c40a69746d8c912c6f158job_title ON default USING GSI
[2022-07-01 13:45:14,567] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee1cbe881e15c40a69746d8c912c6f158job_title+ON+default+USING+GSI
[2022-07-01 13:45:14,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.128771ms
[2022-07-01 13:45:14,617] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee1cbe881e15c40a69746d8c912c6f158job_title'
[2022-07-01 13:45:14,620] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee1cbe881e15c40a69746d8c912c6f158job_title%27
[2022-07-01 13:45:14,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.592445ms
[2022-07-01 13:45:14,624] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '646ff3f2-18a5-41e2-b562-489b51d34d48', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.592445ms', 'executionTime': '1.531877ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:45:14,636] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:45:14,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:14,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:15,335] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:15,343] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:45:15,343] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:45:15,449] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:45:15,463] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:45:15,467] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:15,467] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:16,143] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:16,151] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:45:16,151] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:45:16,254] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:45:16,262] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:45:16,262] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:45:16,262] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:45:16,264] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:45:16,267] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:45:16,274] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.869429ms
[2022-07-01 13:45:16,277] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:45:16,279] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:45:16,331] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.278421ms
[2022-07-01 13:45:16,357] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:45:16,358] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.14406112625494, 'mem_free': 13052370944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:45:16,358] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:45:16,364] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:16,365] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:17,027] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:17,033] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:17,033] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:17,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:17,875] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:17,875] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:18,987] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:18,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:18,996] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:20,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:27,246] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:45:27,272] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:45:28,478] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:45:28,481] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:45:28,481] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:45:28,488] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:45:28,495] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:45:28,502] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:45:28,504] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:45:28,512] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:45:28,513] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:45:28,515] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:28,528] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:45:28,528] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:28,532] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:45:28,535] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:45:28,535] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:28,540] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:45:28,543] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:45:28,544] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:28,547] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:45:28,550] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:45:28,550] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:28,553] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:45:28,553] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-07-01 13:45:28,553] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:45:28,554] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 91.289s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_7'}
[2022-07-01 13:45:28,652] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:28,653] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:29,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:29,262] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:45:29,271] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:45:29,271] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-07-01 13:45:29,272] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:45:29,657] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:45:29,661] - [task:161] INFO -  {'uptime': '549', 'memoryTotal': 15466930176, 'memoryFree': 13184851968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:45:29,663] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:45:29,663] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:45:29,664] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:45:29,675] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:45:29,692] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:45:29,693] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:45:29,695] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:45:29,696] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:45:29,696] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:45:29,697] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:45:29,699] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:45:29,705] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:29,706] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:30,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:30,325] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:45:30,426] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:45:30,427] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:45:30,432] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:30,435] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:30,441] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:45:30,468] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:45:30,472] - [task:161] INFO -  {'uptime': '549', 'memoryTotal': 15466930176, 'memoryFree': 13184098304, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:45:30,475] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:45:30,481] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:45:30,481] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:45:30,483] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:45:30,487] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:30,487] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:31,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:31,160] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:45:31,261] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:45:31,262] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:45:31,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:31,270] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:31,275] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:45:31,294] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:45:31,298] - [task:161] INFO -  {'uptime': '544', 'memoryTotal': 15466930176, 'memoryFree': 13184086016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:45:31,300] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:45:31,304] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:45:31,305] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:45:31,307] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:45:31,315] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:31,315] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:31,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:31,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:45:32,000] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:45:32,001] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:45:32,006] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:32,008] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:32,014] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:45:32,035] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:45:32,038] - [task:161] INFO -  {'uptime': '544', 'memoryTotal': 15466930176, 'memoryFree': 13183758336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:45:32,041] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:45:32,044] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:45:32,045] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:45:32,048] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:45:32,053] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:32,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:32,646] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:32,647] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:45:32,742] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:45:32,743] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:45:32,749] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:32,752] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:45:32,757] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:45:32,776] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:45:32,887] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:45:37,891] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:45:37,905] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:45:37,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:37,910] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:38,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:38,534] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:45:38,632] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:45:38,633] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:45:38,633] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:45:38,792] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:45:38,822] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:45:38,823] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:45:39,397] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:39,440] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:39,484] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:45:39,488] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-01 13:45:39,495] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:39,496] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:40,141] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:40,146] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:40,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:41,019] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:41,032] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:41,032] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:42,084] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:42,094] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:42,094] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:43,167] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:49,516] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:45:49,516] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.04387421465669, 'mem_free': 13196046336, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:45:49,516] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:45:49,517] - [basetestcase:467] INFO - Time to execute basesetup : 20.866618156433105
[2022-07-01 13:45:49,521] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:45:49,521] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:45:49,528] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:45:49,528] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:45:49,534] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:45:49,535] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:45:49,541] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:45:49,541] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:45:49,545] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:45:49,558] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:45:49,559] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:45:49,559] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:45:54,571] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:45:54,576] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:45:54,576] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:45:55,220] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:45:56,381] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:45:56,406] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:45:58,922] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:45:58,952] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:45:58,955] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:45:58,955] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:46:28,979] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:46:28,985] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:46:28,987] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:46:29,008] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.244116ms
[2022-07-01 13:46:29,009] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cb9d8030-9d71-4501-ba12-4d909611845b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.244116ms', 'executionTime': '19.166812ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:46:29,009] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:46:29,011] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:46:29,013] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:46:29,381] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 365.520948ms
[2022-07-01 13:46:29,381] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:46:29,390] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:46:29,407] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:46:29,416] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.564254ms
[2022-07-01 13:46:29,458] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:46:29,462] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:46:29,482] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:46:29,483] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:46:29,497] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:46:29,498] - [base_gsi:326] INFO - []
[2022-07-01 13:46:30,453] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef95ab81183544ed4836fb1f83c30871bjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:46:30,455] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef95ab81183544ed4836fb1f83c30871bjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:46:30,516] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.285745ms
[2022-07-01 13:46:30,524] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef95ab81183544ed4836fb1f83c30871bjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:46:30,529] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef95ab81183544ed4836fb1f83c30871bjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:46:30,601] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.887744ms
[2022-07-01 13:46:30,601] - [base_gsi:282] INFO - BUILD INDEX on default(employeef95ab81183544ed4836fb1f83c30871bjob_title,employeef95ab81183544ed4836fb1f83c30871bjoin_yr) USING GSI
[2022-07-01 13:46:31,608] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeef95ab81183544ed4836fb1f83c30871bjob_title,employeef95ab81183544ed4836fb1f83c30871bjoin_yr) USING GSI
[2022-07-01 13:46:31,610] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeef95ab81183544ed4836fb1f83c30871bjob_title%2Cemployeef95ab81183544ed4836fb1f83c30871bjoin_yr%29+USING+GSI
[2022-07-01 13:46:31,649] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.077962ms
[2022-07-01 13:46:32,656] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjob_title'
[2022-07-01 13:46:32,658] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjob_title%27
[2022-07-01 13:46:32,668] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.500743ms
[2022-07-01 13:46:33,674] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjob_title'
[2022-07-01 13:46:33,677] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjob_title%27
[2022-07-01 13:46:33,686] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.347484ms
[2022-07-01 13:46:33,688] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjoin_yr'
[2022-07-01 13:46:33,690] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjoin_yr%27
[2022-07-01 13:46:33,693] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.136344ms
[2022-07-01 13:46:34,700] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjob_title'
[2022-07-01 13:46:34,702] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjob_title%27
[2022-07-01 13:46:34,712] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.222157ms
[2022-07-01 13:46:34,714] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef95ab81183544ed4836fb1f83c30871bjob_title ON default USING GSI
[2022-07-01 13:46:34,716] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef95ab81183544ed4836fb1f83c30871bjob_title+ON+default+USING+GSI
[2022-07-01 13:46:34,770] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.752003ms
[2022-07-01 13:46:34,779] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjoin_yr'
[2022-07-01 13:46:34,784] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjoin_yr%27
[2022-07-01 13:46:34,787] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.177622ms
[2022-07-01 13:46:34,790] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef95ab81183544ed4836fb1f83c30871bjoin_yr ON default USING GSI
[2022-07-01 13:46:34,792] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef95ab81183544ed4836fb1f83c30871bjoin_yr+ON+default+USING+GSI
[2022-07-01 13:46:34,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.563056ms
[2022-07-01 13:46:34,839] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjob_title'
[2022-07-01 13:46:34,842] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjob_title%27
[2022-07-01 13:46:34,849] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.695939ms
[2022-07-01 13:46:34,849] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bcca4cc3-f567-4b0f-86ea-f9a64c9a0eb2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.695939ms', 'executionTime': '5.636707ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:46:34,852] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef95ab81183544ed4836fb1f83c30871bjoin_yr'
[2022-07-01 13:46:34,854] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef95ab81183544ed4836fb1f83c30871bjoin_yr%27
[2022-07-01 13:46:34,856] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 978.466µs
[2022-07-01 13:46:34,856] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1d7f8972-46a6-4770-8b1b-c5729b7c392e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '978.466µs', 'executionTime': '943.275µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:46:34,866] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:46:34,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:34,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:35,519] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:35,527] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:46:35,527] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:46:35,628] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:46:35,636] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:46:35,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:35,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:36,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:36,278] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:46:36,279] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:46:36,386] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:46:36,394] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:46:36,394] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:46:36,395] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:46:36,397] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:46:36,399] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:46:36,409] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.569937ms
[2022-07-01 13:46:36,411] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:46:36,413] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:46:36,459] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.772178ms
[2022-07-01 13:46:36,476] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:46:36,476] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.864080296961805, 'mem_free': 13089353728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:46:36,476] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:46:36,481] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:36,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:37,080] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:37,085] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:37,086] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:37,692] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:37,698] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:37,698] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:38,829] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:38,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:38,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:39,946] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:46,905] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-07-01 13:46:46,933] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:46:47,248] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:46:47,251] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:46:47,252] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:46:47,257] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:46:47,264] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:46:47,271] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:46:47,272] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:46:47,281] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:46:47,282] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:46:47,284] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,296] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:46:47,296] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,300] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:46:47,304] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:46:47,304] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,308] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:46:47,310] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:46:47,310] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,314] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:46:47,316] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:46:47,316] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,319] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:46:47,319] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-07-01 13:46:47,319] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:46:47,320] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 78.675s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_8'}
[2022-07-01 13:46:47,361] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:47,361] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:47,958] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:47,964] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:46:47,972] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:46:47,972] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-07-01 13:46:47,973] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:46:48,366] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:46:48,370] - [task:161] INFO -  {'uptime': '628', 'memoryTotal': 15466930176, 'memoryFree': 13204955136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:46:48,372] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:46:48,372] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:46:48,372] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:46:48,382] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:46:48,394] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:46:48,395] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:46:48,397] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:46:48,398] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:46:48,398] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:46:48,399] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:46:48,401] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:46:48,405] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:48,407] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:49,011] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:49,012] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:46:49,107] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:46:49,109] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:46:49,113] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:49,116] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:49,122] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:46:49,155] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:46:49,158] - [task:161] INFO -  {'uptime': '624', 'memoryTotal': 15466930176, 'memoryFree': 13204684800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:46:49,160] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:46:49,165] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:46:49,165] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:46:49,168] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:46:49,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:49,172] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:49,778] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:49,779] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:46:49,871] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:46:49,873] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:46:49,878] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:49,881] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:49,887] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:46:49,909] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:46:49,911] - [task:161] INFO -  {'uptime': '625', 'memoryTotal': 15466930176, 'memoryFree': 13204807680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:46:49,914] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:46:49,918] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:46:49,918] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:46:49,921] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:46:49,927] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:49,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:50,520] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:50,521] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:46:50,623] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:46:50,624] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:46:50,629] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:50,632] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:50,637] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:46:50,656] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:46:50,659] - [task:161] INFO -  {'uptime': '625', 'memoryTotal': 15466930176, 'memoryFree': 13207691264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:46:50,662] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:46:50,668] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:46:50,669] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:46:50,671] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:46:50,675] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:50,675] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:51,297] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:51,298] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:46:51,391] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:46:51,392] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:46:51,397] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:51,399] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:46:51,405] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:46:51,419] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:46:51,552] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:46:56,557] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:46:56,569] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:46:56,574] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:56,574] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:57,185] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:57,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:46:57,278] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:46:57,279] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:46:57,279] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:46:57,437] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:46:57,467] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:46:57,467] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:46:58,032] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:46:58,074] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:46:58,855] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:46:58,858] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-01 13:46:58,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:58,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:46:59,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:46:59,538] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:46:59,538] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:00,473] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:00,482] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:00,482] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:01,590] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:01,599] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:01,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:02,691] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:09,182] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:47:09,183] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.1317612017972, 'mem_free': 13211992064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:47:09,183] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:47:09,183] - [basetestcase:467] INFO - Time to execute basesetup : 21.824553728103638
[2022-07-01 13:47:09,188] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:47:09,188] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:47:09,194] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:47:09,195] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:47:09,200] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:47:09,201] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:47:09,207] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:47:09,208] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:47:09,211] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:09,222] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:47:09,222] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:47:09,222] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:47:14,234] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:47:14,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:14,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:14,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:15,918] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:47:15,943] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:47:18,578] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:47:18,620] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:18,620] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:47:18,621] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:47:48,645] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:47:48,651] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:47:48,653] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:47:48,674] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.700958ms
[2022-07-01 13:47:48,674] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c8ad296d-703c-4a49-8bbf-514b35513c12', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.700958ms', 'executionTime': '18.626212ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:47:48,674] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:47:48,677] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:47:48,679] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:47:49,087] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 406.236454ms
[2022-07-01 13:47:49,087] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:47:49,098] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:47:49,103] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:47:49,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.61368ms
[2022-07-01 13:47:49,140] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:49,144] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:47:49,168] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:47:49,168] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:47:49,190] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:47:49,195] - [base_gsi:326] INFO - []
[2022-07-01 13:47:50,141] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec39a13a2067840e6b753929623889abajob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:47:50,143] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec39a13a2067840e6b753929623889abajob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:47:50,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.841524ms
[2022-07-01 13:47:50,193] - [base_gsi:282] INFO - BUILD INDEX on default(employeec39a13a2067840e6b753929623889abajob_title_join_yr) USING GSI
[2022-07-01 13:47:51,198] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec39a13a2067840e6b753929623889abajob_title_join_yr) USING GSI
[2022-07-01 13:47:51,201] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec39a13a2067840e6b753929623889abajob_title_join_yr%29+USING+GSI
[2022-07-01 13:47:51,222] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.289093ms
[2022-07-01 13:47:52,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec39a13a2067840e6b753929623889abajob_title_join_yr'
[2022-07-01 13:47:52,232] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec39a13a2067840e6b753929623889abajob_title_join_yr%27
[2022-07-01 13:47:52,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.002617ms
[2022-07-01 13:47:53,248] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec39a13a2067840e6b753929623889abajob_title_join_yr'
[2022-07-01 13:47:53,251] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec39a13a2067840e6b753929623889abajob_title_join_yr%27
[2022-07-01 13:47:53,260] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.447571ms
[2022-07-01 13:47:54,266] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec39a13a2067840e6b753929623889abajob_title_join_yr'
[2022-07-01 13:47:54,268] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec39a13a2067840e6b753929623889abajob_title_join_yr%27
[2022-07-01 13:47:54,277] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.704939ms
[2022-07-01 13:47:54,279] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec39a13a2067840e6b753929623889abajob_title_join_yr ON default USING GSI
[2022-07-01 13:47:54,281] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec39a13a2067840e6b753929623889abajob_title_join_yr+ON+default+USING+GSI
[2022-07-01 13:47:54,317] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.573727ms
[2022-07-01 13:47:54,324] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec39a13a2067840e6b753929623889abajob_title_join_yr'
[2022-07-01 13:47:54,328] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec39a13a2067840e6b753929623889abajob_title_join_yr%27
[2022-07-01 13:47:54,331] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.572612ms
[2022-07-01 13:47:54,332] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '624079ab-2485-4658-b52d-9e74961b8d60', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.572612ms', 'executionTime': '1.520124ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:47:54,340] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:54,344] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:54,344] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:54,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:54,998] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:47:54,998] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:47:55,102] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:47:55,110] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:55,113] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:55,113] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:55,751] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:55,762] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:47:55,762] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:47:55,883] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:47:55,891] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:47:55,891] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:47:55,891] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:47:55,893] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:47:55,896] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:47:55,903] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.02822ms
[2022-07-01 13:47:55,905] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:47:55,908] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:47:55,957] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.606085ms
[2022-07-01 13:47:55,972] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:47:55,972] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.819671706568458, 'mem_free': 13081174016, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:47:55,972] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:47:55,981] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:55,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:56,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:56,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:56,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:57,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:57,260] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:57,260] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:58,348] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:47:58,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:47:58,360] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:47:59,468] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:06,730] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-07-01 13:48:06,758] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:48:07,889] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:48:07,892] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:48:07,892] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:48:07,899] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:48:07,906] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:48:07,913] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:48:07,914] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:48:07,924] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:48:07,925] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:48:07,928] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:07,942] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:48:07,942] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:07,951] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:48:07,955] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:48:07,956] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:07,959] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:48:07,961] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:48:07,961] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:07,965] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:48:07,967] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:48:07,967] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:07,970] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:48:07,971] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-07-01 13:48:07,971] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:48:07,971] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 80.618s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_9'}
[2022-07-01 13:48:08,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:08,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:08,702] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:08,708] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:48:08,717] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:48:08,717] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-07-01 13:48:08,718] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:48:09,021] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:48:09,025] - [task:161] INFO -  {'uptime': '708', 'memoryTotal': 15466930176, 'memoryFree': 13201051648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:48:09,027] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:48:09,027] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:48:09,028] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:48:09,038] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:48:09,050] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:48:09,051] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:48:09,053] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:48:09,054] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:48:09,054] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:48:09,055] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:48:09,056] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:48:09,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:09,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:09,700] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:09,701] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:48:09,795] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:48:09,797] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:48:09,801] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:09,804] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:09,809] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:48:09,835] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:48:09,839] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13200867328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:48:09,841] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:48:09,846] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:48:09,846] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:48:09,849] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:48:09,852] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:09,853] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:10,490] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:10,491] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:48:10,592] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:48:10,593] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:48:10,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:10,601] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:10,606] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:48:10,625] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:48:10,629] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13201383424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:48:10,631] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:48:10,636] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:48:10,636] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:48:10,639] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:48:10,645] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:10,645] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:11,296] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:11,297] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:48:11,396] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:48:11,397] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:48:11,402] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:11,404] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:11,410] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:48:11,428] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:48:11,431] - [task:161] INFO -  {'uptime': '705', 'memoryTotal': 15466930176, 'memoryFree': 13200887808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:48:11,433] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:48:11,439] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:48:11,440] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:48:11,442] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:48:11,446] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:11,447] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:12,084] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:12,085] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:48:12,179] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:48:12,181] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:48:12,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:12,189] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:48:12,193] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:48:12,209] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:48:12,310] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:48:17,312] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:48:17,323] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:48:17,330] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:17,330] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:17,973] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:17,974] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:48:18,072] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:48:18,074] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:48:18,074] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:48:18,226] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:48:18,259] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:48:18,259] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:48:18,822] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:48:19,049] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:48:19,545] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:48:19,548] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-01 13:48:19,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:19,556] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:20,254] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:20,259] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:20,259] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:21,272] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:21,281] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:21,281] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:22,399] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:22,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:22,408] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:23,521] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:30,150] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:48:30,150] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.30848901797462, 'mem_free': 13197045760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:48:30,150] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:48:30,151] - [basetestcase:467] INFO - Time to execute basesetup : 22.13647985458374
[2022-07-01 13:48:30,155] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:48:30,155] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:48:30,161] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:48:30,161] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:48:30,167] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:48:30,167] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:48:30,173] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:48:30,174] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:48:30,178] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:48:30,188] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:48:30,188] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:48:30,188] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:48:35,201] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:48:35,204] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:48:35,204] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:48:35,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:48:36,910] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:48:36,935] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:48:39,549] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:48:39,586] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:48:39,587] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:48:39,587] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:49:09,603] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:49:09,608] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:49:09,610] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:49:09,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.377168ms
[2022-07-01 13:49:09,632] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f944fc71-8084-4257-a076-a8501f00790b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.377168ms', 'executionTime': '19.298711ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:49:09,632] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:49:09,634] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:49:09,637] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:49:10,044] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 405.875487ms
[2022-07-01 13:49:10,045] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:49:10,054] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:49:10,057] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:49:10,080] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.294156ms
[2022-07-01 13:49:10,103] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:10,108] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:49:10,126] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:49:10,126] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:49:10,138] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:49:10,148] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:11,104] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee163de8714bcf4f51ac782e53466081a5join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:49:11,106] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee163de8714bcf4f51ac782e53466081a5join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:49:11,162] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.414044ms
[2022-07-01 13:49:11,163] - [base_gsi:282] INFO - BUILD INDEX on default(`employee163de8714bcf4f51ac782e53466081a5join_yr`) USING GSI
[2022-07-01 13:49:12,168] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee163de8714bcf4f51ac782e53466081a5join_yr`) USING GSI
[2022-07-01 13:49:12,171] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee163de8714bcf4f51ac782e53466081a5join_yr%60%29+USING+GSI
[2022-07-01 13:49:12,200] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.527223ms
[2022-07-01 13:49:12,211] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee163de8714bcf4f51ac782e53466081a5join_yr'
[2022-07-01 13:49:12,215] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee163de8714bcf4f51ac782e53466081a5join_yr%27
[2022-07-01 13:49:12,229] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 12.38529ms
[2022-07-01 13:49:13,236] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee163de8714bcf4f51ac782e53466081a5join_yr'
[2022-07-01 13:49:13,238] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee163de8714bcf4f51ac782e53466081a5join_yr%27
[2022-07-01 13:49:13,247] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.648864ms
[2022-07-01 13:49:14,254] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee163de8714bcf4f51ac782e53466081a5join_yr'
[2022-07-01 13:49:14,256] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee163de8714bcf4f51ac782e53466081a5join_yr%27
[2022-07-01 13:49:14,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.900558ms
[2022-07-01 13:49:15,271] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:49:15,273] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 13:49:15,278] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.776517ms
[2022-07-01 13:49:15,278] - [base_gsi:504] INFO - {'requestID': 'abbfb433-e7b4-460c-a69b-1b53b27de2ed', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee163de8714bcf4f51ac782e53466081a5join_yr', 'index_id': '83dafdaf790232a9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.776517ms', 'executionTime': '2.695031ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 13:49:15,278] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:49:15,279] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:49:15,279] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:49:15,279] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:49:15,280] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:49:15,281] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:49:15,281] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:49:15,281] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:49:15,342] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:49:15,357] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:49:15,359] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 13:49:15,487] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 112.337237ms
[2022-07-01 13:49:15,487] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:49:15,488] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:49:17,700] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:17,702] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee163de8714bcf4f51ac782e53466081a5join_yr'
[2022-07-01 13:49:17,704] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee163de8714bcf4f51ac782e53466081a5join_yr%27
[2022-07-01 13:49:17,710] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.353581ms
[2022-07-01 13:49:17,710] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '325d3c77-83da-4b33-94da-8e90bbb22c5b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.353581ms', 'executionTime': '4.264396ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:49:17,718] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:17,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:17,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:18,457] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:18,466] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:49:18,467] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:49:18,627] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:49:18,635] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:18,638] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:18,638] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:19,327] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:19,338] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:49:19,338] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:49:19,450] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:49:19,458] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:19,458] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:49:19,458] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:49:19,460] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:49:19,463] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:49:19,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.999946ms
[2022-07-01 13:49:19,472] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '34610211-9f1b-479b-a3f8-022a620fcc7f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.999946ms', 'executionTime': '6.914988ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:49:19,477] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:49:19,477] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.19741175042469, 'mem_free': 13025927168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:49:19,477] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:49:19,483] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:19,483] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:20,181] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:20,186] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:20,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:21,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:21,360] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:21,360] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:22,542] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:22,550] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:22,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:23,744] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:30,686] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-07-01 13:49:30,701] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:49:30,707] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:49:30,713] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:49:30,718] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:49:30,719] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:49:30,725] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:49:30,726] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:49:30,728] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:30,741] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:49:30,741] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:30,744] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:49:30,746] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:49:30,747] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:30,750] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:49:30,752] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:49:30,752] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:30,755] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:49:30,757] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:49:30,757] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:30,760] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:49:30,761] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-07-01 13:49:30,761] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:49:30,762] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 82.753s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_10'}
[2022-07-01 13:49:30,805] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:30,806] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:31,482] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:31,488] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:49:31,496] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:49:31,496] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-07-01 13:49:31,497] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:49:31,809] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:49:31,812] - [task:161] INFO -  {'uptime': '788', 'memoryTotal': 15466930176, 'memoryFree': 13186801664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:49:31,814] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:49:31,814] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:49:31,815] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:49:31,821] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:49:31,832] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:49:31,833] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:49:31,835] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:49:31,838] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:49:31,840] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:49:31,841] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:49:31,842] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:49:31,845] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:31,845] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:32,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:32,494] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:49:32,591] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:49:32,593] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:49:32,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:32,601] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:32,606] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:49:32,637] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:49:32,640] - [task:161] INFO -  {'uptime': '790', 'memoryTotal': 15466930176, 'memoryFree': 13185650688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:49:32,642] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:49:32,647] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:49:32,647] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:49:32,649] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:49:32,656] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:32,657] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:33,333] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:33,334] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:49:33,442] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:49:33,443] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:49:33,448] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:33,450] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:33,455] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:49:33,475] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:49:33,479] - [task:161] INFO -  {'uptime': '785', 'memoryTotal': 15466930176, 'memoryFree': 13185269760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:49:33,481] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:49:33,485] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:49:33,485] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:49:33,488] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:49:33,492] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:33,492] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:34,172] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:34,173] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:49:34,279] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:49:34,280] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:49:34,285] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:34,287] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:34,293] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:49:34,312] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:49:34,315] - [task:161] INFO -  {'uptime': '785', 'memoryTotal': 15466930176, 'memoryFree': 13186703360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:49:34,317] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:49:34,323] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:49:34,323] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:49:34,326] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:49:34,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:34,329] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:34,996] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:34,997] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:49:35,101] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:49:35,102] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:49:35,106] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:35,108] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:49:35,114] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:49:35,137] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:49:35,242] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:49:40,246] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:49:40,258] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:49:40,261] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:40,261] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:40,969] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:40,970] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:49:41,073] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:49:41,074] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:49:41,074] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:49:41,152] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:49:41,184] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:49:41,184] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:49:41,949] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:49:41,992] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:49:42,049] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:49:42,052] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-07-01 13:49:42,059] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:42,059] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:42,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:42,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:42,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:43,799] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:43,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:43,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:44,963] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:44,975] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:44,975] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:46,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:52,944] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:49:52,944] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 27.2049011858498, 'mem_free': 13180026880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:49:52,944] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:49:52,944] - [basetestcase:467] INFO - Time to execute basesetup : 22.14177179336548
[2022-07-01 13:49:52,949] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:49:52,950] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:49:52,956] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:49:52,957] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:49:52,963] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:49:52,963] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:49:52,969] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:49:52,970] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:49:52,974] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:49:52,987] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:49:52,987] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:49:52,987] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:49:57,996] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:49:57,999] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:49:57,999] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:49:58,642] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:49:59,707] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:49:59,732] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:50:02,443] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:50:02,472] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:02,472] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:50:02,473] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:50:32,502] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:50:32,507] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:50:32,509] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:50:32,529] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 17.87377ms
[2022-07-01 13:50:32,529] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0f3175aa-2feb-4946-b27e-0fcb36211ae8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '17.87377ms', 'executionTime': '17.801602ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:50:32,529] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:50:32,532] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:50:32,534] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:50:32,950] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 414.345483ms
[2022-07-01 13:50:32,951] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:50:32,962] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:50:32,967] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:50:32,976] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.779234ms
[2022-07-01 13:50:33,011] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:33,021] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:50:33,049] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:50:33,050] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:50:33,061] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:50:33,074] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:34,001] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4119f62902fc42898230e19f2abe8659join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:50:34,003] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4119f62902fc42898230e19f2abe8659join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:50:34,059] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.928513ms
[2022-07-01 13:50:34,059] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4119f62902fc42898230e19f2abe8659join_yr`) USING GSI
[2022-07-01 13:50:35,065] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4119f62902fc42898230e19f2abe8659join_yr`) USING GSI
[2022-07-01 13:50:35,067] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4119f62902fc42898230e19f2abe8659join_yr%60%29+USING+GSI
[2022-07-01 13:50:35,092] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.757197ms
[2022-07-01 13:50:35,110] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4119f62902fc42898230e19f2abe8659join_yr'
[2022-07-01 13:50:35,113] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4119f62902fc42898230e19f2abe8659join_yr%27
[2022-07-01 13:50:35,123] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.784339ms
[2022-07-01 13:50:36,129] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4119f62902fc42898230e19f2abe8659join_yr'
[2022-07-01 13:50:36,131] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4119f62902fc42898230e19f2abe8659join_yr%27
[2022-07-01 13:50:36,139] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.854855ms
[2022-07-01 13:50:37,145] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4119f62902fc42898230e19f2abe8659join_yr'
[2022-07-01 13:50:37,147] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4119f62902fc42898230e19f2abe8659join_yr%27
[2022-07-01 13:50:37,154] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.38989ms
[2022-07-01 13:50:38,160] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:50:38,163] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 13:50:38,167] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.883798ms
[2022-07-01 13:50:38,167] - [base_gsi:504] INFO - {'requestID': '939ff201-9563-472d-acb9-10a6be6bb0f3', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4119f62902fc42898230e19f2abe8659join_yr', 'index_id': 'cb9cdc0ebf249aec', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.883798ms', 'executionTime': '2.79219ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 13:50:38,168] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:50:38,168] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:38,168] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:50:38,169] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:50:38,169] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:38,169] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:38,169] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:38,170] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:38,230] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:38,244] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:50:38,246] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 13:50:38,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.634992ms
[2022-07-01 13:50:38,357] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:50:38,357] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:50:39,525] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-07-01 13:50:39,557] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-07-01 13:50:39,563] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:50:39,565] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 13:50:39,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.302078ms
[2022-07-01 13:50:39,569] - [base_gsi:504] INFO - {'requestID': '61e3b308-e50f-4d36-8bcd-e118265fee1d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4119f62902fc42898230e19f2abe8659join_yr', 'index_id': 'cb9cdc0ebf249aec', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.302078ms', 'executionTime': '2.21871ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-07-01 13:50:39,570] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:50:39,570] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:39,570] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:50:39,570] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:50:39,571] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:39,571] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:39,571] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:39,571] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:50:39,628] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:50:39,643] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:50:39,645] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 13:50:39,751] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.503308ms
[2022-07-01 13:50:39,751] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:50:39,752] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:50:40,941] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4119f62902fc42898230e19f2abe8659join_yr ON default USING GSI
[2022-07-01 13:50:40,944] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4119f62902fc42898230e19f2abe8659join_yr+ON+default+USING+GSI
[2022-07-01 13:50:40,996] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.318136ms
[2022-07-01 13:50:41,004] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4119f62902fc42898230e19f2abe8659join_yr'
[2022-07-01 13:50:41,008] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4119f62902fc42898230e19f2abe8659join_yr%27
[2022-07-01 13:50:41,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.853794ms
[2022-07-01 13:50:41,016] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '953e5dff-673a-43ff-8931-d747a374df77', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.853794ms', 'executionTime': '5.776871ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:50:41,024] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:41,027] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:41,027] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:41,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:41,793] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:50:41,794] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:50:41,905] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:50:41,913] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:41,916] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:41,916] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:42,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:42,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:50:42,628] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:50:42,742] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:50:42,750] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:50:42,750] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:50:42,750] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:50:42,753] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:50:42,755] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:50:42,765] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.242961ms
[2022-07-01 13:50:42,767] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:50:42,769] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:50:42,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.424248ms
[2022-07-01 13:50:42,838] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:50:42,838] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.90328844605091, 'mem_free': 13019897856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:50:42,839] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:50:42,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:42,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:43,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:43,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:43,558] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:44,381] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:44,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:44,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:45,674] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:45,682] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:45,682] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:46,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:54,405] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-07-01 13:50:54,428] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:50:55,650] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:50:55,653] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:50:55,653] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:50:55,660] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:50:55,665] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:50:55,671] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:50:55,672] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:50:55,680] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:50:55,681] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:50:55,683] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:55,696] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:50:55,696] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:55,699] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:50:55,701] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:50:55,701] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:55,705] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:50:55,708] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:50:55,709] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:55,713] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:50:55,716] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:50:55,716] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:55,720] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:50:55,720] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-07-01 13:50:55,720] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:50:55,721] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 84.924s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_11'}
[2022-07-01 13:50:55,765] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:55,765] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:56,470] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:56,476] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:50:56,485] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:50:56,486] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-07-01 13:50:56,486] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:50:56,769] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:50:56,773] - [task:161] INFO -  {'uptime': '876', 'memoryTotal': 15466930176, 'memoryFree': 13167505408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:50:56,775] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:50:56,776] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:50:56,776] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:50:56,786] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:50:56,791] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:50:56,792] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:50:56,795] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:50:56,796] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:50:56,797] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:50:56,798] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:50:56,801] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:50:56,804] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:56,805] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:57,504] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:57,505] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:50:57,609] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:50:57,610] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:50:57,615] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:57,618] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:57,624] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:50:57,650] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:50:57,654] - [task:161] INFO -  {'uptime': '875', 'memoryTotal': 15466930176, 'memoryFree': 13170757632, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:50:57,656] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:50:57,662] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:50:57,662] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:50:57,664] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:50:57,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:57,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:58,363] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:58,364] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:50:58,467] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:50:58,468] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:50:58,473] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:58,475] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:58,481] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:50:58,500] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:50:58,503] - [task:161] INFO -  {'uptime': '871', 'memoryTotal': 15466930176, 'memoryFree': 13169172480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:50:58,506] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:50:58,510] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:50:58,511] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:50:58,513] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:50:58,519] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:58,519] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:50:59,231] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:50:59,232] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:50:59,336] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:50:59,337] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:50:59,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:59,345] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:50:59,350] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:50:59,367] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:50:59,371] - [task:161] INFO -  {'uptime': '871', 'memoryTotal': 15466930176, 'memoryFree': 13169721344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:50:59,373] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:50:59,377] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:50:59,377] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:50:59,381] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:50:59,387] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:50:59,387] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:00,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:00,087] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:51:00,200] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:51:00,201] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:51:00,206] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:51:00,209] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:51:00,214] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:51:00,229] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:51:00,356] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:51:05,357] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:51:05,371] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:51:05,377] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:05,377] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:06,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:06,139] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:51:06,253] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:51:06,254] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:51:06,255] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:51:07,248] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:51:07,277] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:51:07,278] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:51:07,844] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:51:07,885] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:51:08,562] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:51:08,565] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 13:51:08,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:08,580] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:09,332] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:09,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:09,338] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:10,386] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:10,399] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:10,399] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:11,616] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:11,623] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:11,623] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:12,834] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:20,043] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:51:20,043] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.88193534338091, 'mem_free': 13169971200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:51:20,043] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:51:20,043] - [basetestcase:467] INFO - Time to execute basesetup : 24.281212329864502
[2022-07-01 13:51:20,048] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:51:20,048] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:51:20,054] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:51:20,054] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:51:20,061] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:51:20,061] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:51:20,067] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:51:20,067] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:51:20,071] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:51:20,085] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:51:20,085] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:51:20,086] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:51:25,097] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:51:25,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:51:25,101] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:51:25,799] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:51:26,887] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:51:27,085] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:51:29,631] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:51:29,662] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:51:29,662] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:51:29,662] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:51:59,672] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:51:59,678] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:51:59,680] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:51:59,702] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.476718ms
[2022-07-01 13:51:59,702] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9b4fe0f7-fae9-42da-977c-c3b8c0d59f1a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.476718ms', 'executionTime': '19.383337ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:51:59,702] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:51:59,705] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:51:59,707] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:52:00,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 412.279444ms
[2022-07-01 13:52:00,122] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:52:00,127] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:52:00,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:52:00,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.412384ms
[2022-07-01 13:52:00,175] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:00,180] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:52:00,198] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:52:00,199] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:52:00,209] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:52:00,229] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:01,176] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-07-01 13:52:01,178] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-07-01 13:52:01,249] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.787672ms
[2022-07-01 13:52:01,251] - [base_gsi:282] INFO - BUILD INDEX on default(`employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr`) USING GSI
[2022-07-01 13:52:02,255] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr`) USING GSI
[2022-07-01 13:52:02,258] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%60%29+USING+GSI
[2022-07-01 13:52:02,292] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.877813ms
[2022-07-01 13:52:02,307] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr'
[2022-07-01 13:52:02,313] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%27
[2022-07-01 13:52:02,322] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.199755ms
[2022-07-01 13:52:03,329] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr'
[2022-07-01 13:52:03,331] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%27
[2022-07-01 13:52:03,340] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.916103ms
[2022-07-01 13:52:04,346] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr'
[2022-07-01 13:52:04,348] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%27
[2022-07-01 13:52:04,357] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.641862ms
[2022-07-01 13:52:05,364] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:52:05,366] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-07-01 13:52:05,370] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.71546ms
[2022-07-01 13:52:05,371] - [base_gsi:504] INFO - {'requestID': 'ec3445c7-4205-49c3-a3a7-01303c2e30cb', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr', 'index_id': 'a3946f340bc2401', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.71546ms', 'executionTime': '2.59534ms', 'resultCount': 1, 'resultSize': 889, 'serviceLoad': 6}}
[2022-07-01 13:52:05,371] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-07-01 13:52:05,372] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:52:05,372] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-07-01 13:52:05,372] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-07-01 13:52:05,372] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:52:05,372] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:52:05,373] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:52:05,373] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-07-01 13:52:05,434] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-07-01 13:52:05,450] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-07-01 13:52:05,452] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-07-01 13:52:05,568] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 103.377746ms
[2022-07-01 13:52:05,569] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-07-01 13:52:05,569] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-07-01 13:52:07,705] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-01 13:52:10,312] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:52:10,346] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:52:10,346] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:52:10,909] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:52:10,960] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:52:11,002] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:52:11,005] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-07-01 13:52:13,013] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:52:13,013] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:13,017] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:52:13,021] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:52:13,022] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:13,025] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:52:13,029] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:52:13,029] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:13,032] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:52:13,037] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:52:13,037] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:13,040] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:52:13,045] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:13,047] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr'
[2022-07-01 13:52:13,049] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%27
[2022-07-01 13:52:13,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.961982ms
[2022-07-01 13:52:13,071] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'dbcb6506-a86f-4f6b-922d-ea89b645cfb0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.961982ms', 'executionTime': '19.828117ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:52:13,074] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr'
[2022-07-01 13:52:13,076] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7ea226fd62f0400b96d7d8e2f956fdd8join_yr%27
[2022-07-01 13:52:13,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.20076ms
[2022-07-01 13:52:13,079] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bb737f61-be89-4ed1-8ab0-f9ba9b801ee6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.20076ms', 'executionTime': '1.149377ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:52:13,082] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:13,093] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:13,095] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:13,095] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:13,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:13,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:52:13,896] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:52:14,018] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:52:14,028] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:14,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:14,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:14,800] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:14,808] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:52:14,808] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:52:14,926] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:52:14,934] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:14,934] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:52:14,934] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:52:14,936] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:52:14,939] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:52:14,951] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.16155ms
[2022-07-01 13:52:14,951] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '533d3a8e-619a-4c42-9a3e-9e133d2ec0b4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '10.16155ms', 'executionTime': '10.092701ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:52:14,951] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:52:14,954] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:52:14,956] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:52:14,958] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 897.243µs
[2022-07-01 13:52:14,958] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8cdf62bd-cbb7-4f15-a504-893338631990', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '897.243µs', 'executionTime': '859.49µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:52:14,962] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:52:14,962] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.90388789360659, 'mem_free': 12984840192, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:52:14,962] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:52:14,966] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:14,966] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:15,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:15,694] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:15,695] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:16,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:16,731] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:16,732] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:17,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:17,968] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:17,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:19,212] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:26,724] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 13:52:26,747] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:52:27,028] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:52:27,032] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:52:27,032] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:52:27,038] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:52:27,045] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:52:27,051] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:52:27,052] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:52:27,060] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:52:27,061] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:52:27,063] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,075] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:52:27,076] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,079] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:52:27,082] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:52:27,082] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,088] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:52:27,090] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:52:27,090] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,093] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:52:27,095] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:52:27,096] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,099] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:52:27,099] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-07-01 13:52:27,100] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:52:27,101] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 91.343s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jul-01_13-36-16/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-07-01 13:52:27,111] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:27,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:27,845] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:27,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:52:27,873] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1432-rel-enterprise
[2022-07-01 13:52:27,873] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-07-01 13:52:27,874] - [basetestcase:224] INFO - initializing cluster
[2022-07-01 13:52:28,115] - [task:156] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-07-01 13:52:28,119] - [task:161] INFO -  {'uptime': '968', 'memoryTotal': 15466930176, 'memoryFree': 13150945280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:52:28,121] - [task:200] INFO - quota for index service will be 256 MB
[2022-07-01 13:52:28,121] - [task:202] INFO - set index quota to node 127.0.0.1 
[2022-07-01 13:52:28,122] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-07-01 13:52:28,132] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-07-01 13:52:28,141] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-07-01 13:52:28,142] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-07-01 13:52:28,144] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-07-01 13:52:28,145] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-07-01 13:52:28,151] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-07-01 13:52:28,152] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-07-01 13:52:28,154] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:52:28,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:28,156] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:28,871] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:28,872] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:52:28,979] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:52:28,980] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:52:28,986] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:28,989] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:28,994] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:52:29,024] - [task:156] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-07-01 13:52:29,028] - [task:161] INFO -  {'uptime': '966', 'memoryTotal': 15466930176, 'memoryFree': 13149347840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:52:29,030] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:52:29,034] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-07-01 13:52:29,035] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-07-01 13:52:29,037] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:52:29,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:29,041] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:29,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:29,778] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:52:29,888] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:52:29,889] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:52:29,894] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:29,896] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:29,901] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:52:29,925] - [task:156] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-07-01 13:52:29,928] - [task:161] INFO -  {'uptime': '966', 'memoryTotal': 15466930176, 'memoryFree': 13149888512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:52:29,930] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:52:29,934] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-07-01 13:52:29,934] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-07-01 13:52:29,938] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:52:29,941] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:29,941] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:30,667] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:30,668] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:52:30,781] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:52:30,782] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:52:30,787] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:30,790] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:30,795] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:52:30,819] - [task:156] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-07-01 13:52:30,823] - [task:161] INFO -  {'uptime': '961', 'memoryTotal': 15466930176, 'memoryFree': 13149446144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1432-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-07-01 13:52:30,825] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-07-01 13:52:30,830] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-07-01 13:52:30,830] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-07-01 13:52:30,833] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-07-01 13:52:30,837] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:30,837] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:31,553] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:31,554] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:52:31,662] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:52:31,663] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:52:31,668] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:31,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-07-01 13:52:31,676] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-07-01 13:52:31,691] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-07-01 13:52:31,796] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:52:36,800] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-07-01 13:52:36,814] - [basetestcase:262] INFO - done initializing cluster
[2022-07-01 13:52:36,819] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:36,820] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:37,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:37,609] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-07-01 13:52:37,727] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-07-01 13:52:37,729] - [remote_util:5249] INFO - b'ok'
[2022-07-01 13:52:37,729] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-07-01 13:52:38,712] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-07-01 13:52:38,741] - [on_prem_rest_client:3047] INFO - 0.03 seconds to create bucket default
[2022-07-01 13:52:38,741] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-07-01 13:52:39,304] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:52:39,341] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:52:40,007] - [task:379] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-07-01 13:52:40,012] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-07-01 13:52:40,019] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:40,019] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:40,817] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:40,822] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:40,822] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:41,989] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:41,997] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:41,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:43,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:43,267] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:43,267] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:44,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:51,787] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:52:51,787] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.82341086500955, 'mem_free': 13142405120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:52:51,787] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:52:51,787] - [basetestcase:467] INFO - Time to execute basesetup : 24.67877173423767
[2022-07-01 13:52:51,791] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:52:51,792] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:52:51,797] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:52:51,798] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:52:51,804] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:52:51,804] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:52:51,810] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-07-01 13:52:51,810] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-07-01 13:52:51,814] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:52:51,826] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-07-01 13:52:51,826] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-07-01 13:52:51,826] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-07-01 13:52:56,840] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-07-01 13:52:56,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:52:56,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:52:57,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:52:58,637] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-07-01 13:52:58,662] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-07-01 13:53:01,352] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-07-01 13:53:01,380] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:01,380] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-07-01 13:53:01,380] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-07-01 13:53:31,400] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-07-01 13:53:31,405] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:53:31,407] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:53:31,428] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.185692ms
[2022-07-01 13:53:31,428] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bacb5e09-ae03-4390-af85-ea1924c30624', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.185692ms', 'executionTime': '19.11713ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-07-01 13:53:31,429] - [tuq_helper:758] INFO - Create primary index
[2022-07-01 13:53:31,431] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-07-01 13:53:31,433] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-07-01 13:53:31,825] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 390.313858ms
[2022-07-01 13:53:31,825] - [tuq_helper:760] INFO - Check if index is online
[2022-07-01 13:53:31,831] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:53:31,838] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:53:31,848] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.50646ms
[2022-07-01 13:53:31,877] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:31,881] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-07-01 13:53:31,897] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-07-01 13:53:31,897] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-07-01 13:53:31,912] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-07-01 13:53:31,925] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:31,946] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:31,951] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:31,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:32,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:32,864] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:53:32,864] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-07-01 13:53:32,996] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:53:33,005] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:33,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:33,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:33,790] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:33,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-07-01 13:53:33,798] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-07-01 13:53:33,922] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-07-01 13:53:33,934] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-07-01 13:53:33,934] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-07-01 13:53:33,934] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:53:33,937] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-07-01 13:53:33,939] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-07-01 13:53:33,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 14.462575ms
[2022-07-01 13:53:33,957] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-07-01 13:53:33,959] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-07-01 13:53:34,021] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.543192ms
[2022-07-01 13:53:34,033] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-07-01 13:53:34,033] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.899666838134293, 'mem_free': 12973838336, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-07-01 13:53:34,034] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-07-01 13:53:34,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:34,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:34,852] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:34,857] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:34,857] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:36,036] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:36,043] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:36,044] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:37,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:37,462] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-07-01 13:53:37,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-07-01 13:53:38,832] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-07-01 13:53:46,994] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-07-01 13:53:47,019] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-07-01 13:53:48,173] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-07-01 13:53:48,176] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-07-01 13:53:48,176] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-07-01 13:53:48,183] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:53:48,189] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:53:48,196] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-07-01 13:53:48,196] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-07-01 13:53:48,206] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-07-01 13:53:48,207] - [basetestcase:742] INFO - b'"User was not found."'
[2022-07-01 13:53:48,210] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:53:48,222] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-07-01 13:53:48,222] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:53:48,226] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-07-01 13:53:48,230] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-07-01 13:53:48,230] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:53:48,235] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-07-01 13:53:48,239] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-07-01 13:53:48,239] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:53:48,242] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-07-01 13:53:48,245] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-07-01 13:53:48,245] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-07-01 13:53:48,248] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-07-01 13:53:48,248] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-07-01 13:53:48,249] - [basetestcase:773] INFO - closing all ssh connections
[2022-07-01 13:53:48,249] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 81.178s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 21538 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

FAIL	github.com/couchbase/plasma	3675.330s
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests [build failed]
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests [build failed]
--- FAIL: TestShardMetadata (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
Version: versions-01.07.2022-11.17.cfg
Build Log: make-01.07.2022-11.17.log
Server Log: logs-01.07.2022-11.17.tar.gz

Finished