Building

Started building at 2022/11/25 14:42:45
Using pegged server, 3184 build
Calculating base
Updating mirror
Basing run on 7.5.0-3184 ef82d9906
Updating tree for run 25.11.2022-14.42
query is at 03ee514, changes since last good build: none
gometa is at 7b72c0b, changes since last good build: 
fatal: Invalid revision range 9770c119705b57ca83b224c16544cfdb2f62fecf..HEAD

ns_server is at 1a8c9f5, changes since last good build: none
couchstore is at 82a0bbf, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 702340a, changes since last good build: none
Switching indexing to unstable
indexing is at a24365b, changes since last good build: none
Switching plasma to unstable
plasma is at 01a420e, changes since last good build: 
fatal: Invalid revision range 04301f585e26128c618f5f0a55e66b00c086d2cc..HEAD

Switching nitro to unstable
nitro is at d5f5610, changes since last good build: none
Switching gometa to master
gometa is at 4f94ea5, changes since last good build: 
fatal: Invalid revision range 9770c119705b57ca83b224c16544cfdb2f62fecf..HEAD

Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule 'magma_loader/DocLoader' (https://github.com/couchbaselabs/DocLoader.git) registered for path 'magma_loader/DocLoader'
Cloning into '/opt/build/testrunner/gauntlet'...
Cloning into '/opt/build/testrunner/java_sdk_client'...
Cloning into '/opt/build/testrunner/lib/capellaAPI'...
Cloning into '/opt/build/testrunner/magma_loader/DocLoader'...
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '5dd338995c16ac2f5b187729e549b28862060732'
Submodule path 'lib/capellaAPI': checked out 'e4b44567beeea5ce9390324e46f084b79ea7c5e5'
Submodule path 'magma_loader/DocLoader': checked out '997f514b43e9beef241d0b7e61c8171042d22543'
testrunner is at 207c1fd, changes since last good build: none
Pulling in uncommitted change 182479 at refs/changes/79/182479/17
Total 10 (delta 9), reused 10 (delta 9)
[unstable 2e729fb0] MB-53988: Throttle scan when mutation is falling behind
 Author: jliang00 
 Date: Tue Nov 8 14:54:54 2022 -0800
 5 files changed, 119 insertions(+)
Pulling in uncommitted change 183142 at refs/changes/42/183142/1
Total 33 (delta 27), reused 33 (delta 27)
[unstable d8a4f4b1] MB-54634 Do not process collection drop messages twice
 Author: Varun Velamuri 
 Date: Tue Nov 22 09:52:38 2022 -0800
 1 file changed, 14 insertions(+), 1 deletion(-)
Pulling in uncommitted change 183197 at refs/changes/97/183197/2
Total 5 (delta 2), reused 2 (delta 2)
[unstable 466870fa] MB-54438: Use new plasma writer throttle and SMR transaction API
 Author: akhilmd 
 Date: Wed Nov 23 23:51:54 2022 +0530
 1 file changed, 22 insertions(+), 8 deletions(-)
Pulling in uncommitted change 183196 at refs/changes/96/183196/3
Total 3 (delta 1), reused 1 (delta 1)
[unstable 2096fa9] MB-54438: Separate writer throttling from SMR transaction begin
 Author: akhilmd 
 Date: Wed Nov 23 23:49:19 2022 +0530
 1 file changed, 21 insertions(+), 2 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/11/25 15:36:03
Testing mode: sanity,unit,functional,serverless,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Nov 25 15:41:31 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Nov 25 15:45:25 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Nov 25 15:46:14 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Nov 25 15:47:40 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Nov 25 15:48:34 b" 'stop_on_failure': 'True'}"
Nov 25 15:48:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Nov 25 15:48:34 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_1'}"
Nov 25 15:48:34 b'-->result: '
Nov 25 15:48:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Nov 25 15:48:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Nov 25 15:48:34 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_2'}"
Nov 25 15:48:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 15:48:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Nov 25 15:48:34 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_3'}"
Nov 25 15:48:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 15:48:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 15:48:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Nov 25 15:48:34 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_4'}"
Nov 25 15:48:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 15:48:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 15:48:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Nov 25 15:57:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Nov 25 15:58:42 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Nov 25 16:02:28 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Nov 25 16:06:27 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Nov 25 16:09:12 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Nov 25 16:09:12 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_5'}"
Nov 25 16:09:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Nov 25 16:09:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Nov 25 16:09:12 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_6'}"
Nov 25 16:09:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Nov 25 16:09:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Nov 25 16:09:12 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_7'}"
Nov 25 16:09:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Nov 25 16:09:12 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Nov 25 16:09:12 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_15-36-27/test_8'}"
Nov 25 16:09:12 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Nov 25 16:09:12 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Nov 25 16:09:12 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Nov 25 16:09:12 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Nov 25 16:09:13 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Nov 25 16:09:13 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Nov 25 16:09:13 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Nov 25 16:09:13 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Nov 25 16:09:13 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Nov 25 16:09:13 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Nov 25 16:09:13 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Nov 25 16:09:13 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.02s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000 items took 16.270817ms -> 983355.6606284736 items/s conflicts 0
--- PASS: TestInsertPerf (0.02s)
=== RUN   TestGetPerf
16000 items took 6.435361ms -> 2.486263008399995e+06 items/s
--- PASS: TestGetPerf (0.01s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3450,
"memory_used":            45520,
"node_allocs":            1000,
"node_frees":             0,
"level_node_distribution":{
"level0": 747,
"level1": 181,
"level2": 56,
"level3": 13,
"level4": 2,
"level5": 1,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [105 161 346 379 434 523 713]
No of items in each range [105 56 185 33 55 89 190 287]
--- PASS: TestGetRangeSplitItems (0.00s)
=== RUN   TestBuilder
{
"node_count":             50000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3368,
"memory_used":            2269408,
"node_allocs":            50000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37380,
"level1": 9466,
"level2": 2370,
"level3": 578,
"level4": 152,
"level5": 40,
"level6": 9,
"level7": 4,
"level8": 1,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 8.105168ms to build 50000 items, 6.1689035e+06 items/sec
Took 1.919345ms to iterate 50000 items
--- PASS: TestBuilder (0.01s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/nitro/skiplist	0.071s
=== RUN   TestZstdSimple
--- PASS: TestZstdSimple (0.00s)
=== RUN   TestZstdCompressBound
--- PASS: TestZstdCompressBound (3.07s)
=== RUN   TestZstdErrors
--- PASS: TestZstdErrors (0.00s)
=== RUN   TestZstdCompressLevels
--- PASS: TestZstdCompressLevels (0.75s)
=== RUN   TestZstdEmptySrc
--- PASS: TestZstdEmptySrc (0.00s)
=== RUN   TestZstdLargeSrc
--- PASS: TestZstdLargeSrc (0.00s)
PASS
ok  	github.com/couchbase/plasma/zstd	3.836s
go: downloading github.com/golang/snappy v0.0.4
go: downloading github.com/edsrzf/mmap-go v1.1.0
go: downloading github.com/aws/aws-sdk-go v1.44.105
go: downloading github.com/couchbase/tools-common v0.0.0-20221108111232-74639726fb4d
go: downloading github.com/couchbase/logstats v0.0.0-20220303011129-24ba9753289f
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/stretchr/objx v0.4.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
=== RUN   TestAutoTunerWriteUsageStats
--- PASS: TestAutoTunerWriteUsageStats (10.07s)
=== RUN   TestAutoTunerReadUsageStats
--- PASS: TestAutoTunerReadUsageStats (7.61s)
=== RUN   TestAutoTunerCleanerUsageStats
--- PASS: TestAutoTunerCleanerUsageStats (8.54s)
=== RUN   TestAutoTunerDiskStats
--- PASS: TestAutoTunerDiskStats (2.50s)
=== RUN   TestAutoTunerTargetFragRatio
--- PASS: TestAutoTunerTargetFragRatio (0.00s)
=== RUN   TestAutoTunerExcessUsedSpace
--- PASS: TestAutoTunerExcessUsedSpace (0.00s)
=== RUN   TestAutoTunerUsedSpaceRatio
--- PASS: TestAutoTunerUsedSpaceRatio (0.00s)
=== RUN   TestAutoTunerAdjustFragRatio
--- PASS: TestAutoTunerAdjustFragRatio (0.00s)
=== RUN   TestAutoTuneFlushBufferAdjustMemQuotaSingleShard
--- PASS: TestAutoTuneFlushBufferAdjustMemQuotaSingleShard (16.07s)
=== RUN   TestAutoTuneFlushBufferAdjustMemQuotaManyShards
--- PASS: TestAutoTuneFlushBufferAdjustMemQuotaManyShards (10.72s)
=== RUN   TestAutoTuneFlushBufferRebalanceIdleShards
--- PASS: TestAutoTuneFlushBufferRebalanceIdleShards (9.45s)
=== RUN   TestAutoTuneFlushBufferGetUsedMemory
--- PASS: TestAutoTuneFlushBufferGetUsedMemory (17.34s)
=== RUN   TestBloom
--- PASS: TestBloom (4.34s)
=== RUN   TestBloomDisableEnable
--- PASS: TestBloomDisableEnable (3.40s)
=== RUN   TestBloomDisable
--- PASS: TestBloomDisable (0.04s)
=== RUN   TestBloomFreeDuringLookup
--- PASS: TestBloomFreeDuringLookup (0.03s)
=== RUN   TestBloomRecoveryFreeDuringLookup
--- PASS: TestBloomRecoveryFreeDuringLookup (0.11s)
=== RUN   TestBloomRecoverySwapInLookup
--- PASS: TestBloomRecoverySwapInLookup (0.07s)
=== RUN   TestBloomRecoverySwapOutLookup
--- PASS: TestBloomRecoverySwapOutLookup (0.12s)
=== RUN   TestBloomRecoveryInserts
--- PASS: TestBloomRecoveryInserts (0.07s)
=== RUN   TestBloomRecovery
--- PASS: TestBloomRecovery (0.13s)
=== RUN   TestBloomStats
--- PASS: TestBloomStats (3.33s)
=== RUN   TestBloomStatsRecovery
--- PASS: TestBloomStatsRecovery (0.83s)
=== RUN   TestBloomFilterSimple
--- PASS: TestBloomFilterSimple (0.00s)
=== RUN   TestBloomFilterConcurrent
--- PASS: TestBloomFilterConcurrent (22.06s)
=== RUN   TestBitArrayConcurrent
--- PASS: TestBitArrayConcurrent (1.08s)
=== RUN   TestBloomCapacity
--- PASS: TestBloomCapacity (0.00s)
=== RUN   TestBloomNumHashFuncs
--- PASS: TestBloomNumHashFuncs (0.00s)
=== RUN   TestBloomTestAndAdd
--- PASS: TestBloomTestAndAdd (0.22s)
=== RUN   TestBloomReset
--- PASS: TestBloomReset (0.00s)
=== RUN   TestLFSCopier
--- PASS: TestLFSCopier (0.00s)
=== RUN   TestLFSCopierNumBytes
--- PASS: TestLFSCopierNumBytes (0.01s)
=== RUN   TestSBCopyConcurrent
--- PASS: TestSBCopyConcurrent (0.22s)
=== RUN   TestSBCopyCorrupt
--- PASS: TestSBCopyCorrupt (0.01s)
=== RUN   TestLSSCopyHeadTailSingleSegment
--- PASS: TestLSSCopyHeadTailSingleSegment (0.03s)
=== RUN   TestLSSCopyFullSegments
--- PASS: TestLSSCopyFullSegments (0.57s)
=== RUN   TestLSSCopyPartialSegments
--- PASS: TestLSSCopyPartialSegments (0.06s)
=== RUN   TestLSSCopyHolePunching
--- PASS: TestLSSCopyHolePunching (0.59s)
=== RUN   TestLSSCopyConcurrent
--- PASS: TestLSSCopyConcurrent (0.80s)
=== RUN   TestLSSCopyS3MultipartHeapUsage
--- PASS: TestLSSCopyS3MultipartHeapUsage (0.00s)
=== RUN   TestShardCopySimple
--- PASS: TestShardCopySimple (0.29s)
=== RUN   TestShardCopyWithTimeout
--- PASS: TestShardCopyWithTimeout (0.00s)
=== RUN   TestShardCopyNoRP
--- PASS: TestShardCopyNoRP (0.13s)
=== RUN   TestShardCopyMetadataCorrupted
--- PASS: TestShardCopyMetadataCorrupted (0.03s)
=== RUN   TestShardCopyLSSMetadataCorrupted
--- PASS: TestShardCopyLSSMetadataCorrupted (0.10s)
=== RUN   TestShardCopyBeforeRecovery
--- PASS: TestShardCopyBeforeRecovery (0.00s)
=== RUN   TestShardCopySkipLog
--- PASS: TestShardCopySkipLog (0.67s)
=== RUN   TestShardCopyAddInstance
--- PASS: TestShardCopyAddInstance (1.61s)
=== RUN   TestShardCopyDestroyInstance
--- PASS: TestShardCopyDestroyInstance (0.26s)
=== RUN   TestShardCopyRestoreShard
--- PASS: TestShardCopyRestoreShard (9.43s)
=== RUN   TestShardCopyRestoreManyShards
--- PASS: TestShardCopyRestoreManyShards (5.76s)
=== RUN   TestShardCopyRestoreConcurrentLogCleaning
--- PASS: TestShardCopyRestoreConcurrentLogCleaning (21.96s)
=== RUN   TestShardCopyRestorePartialRollback
--- PASS: TestShardCopyRestorePartialRollback (11.96s)
=== RUN   TestInvalidMVCCRollback
--- PASS: TestInvalidMVCCRollback (0.22s)
=== RUN   TestShardCopyRestoreConcurrentPurges
--- PASS: TestShardCopyRestoreConcurrentPurges (12.50s)
=== RUN   TestShardCopyDuplicateIndex
--- PASS: TestShardCopyDuplicateIndex (0.14s)
=== RUN   TestTenantCopy
--- PASS: TestTenantCopy (3.86s)
=== RUN   TestLockShardAddInstance
--- PASS: TestLockShardAddInstance (0.16s)
=== RUN   TestLockShardAddInstanceMapping
--- PASS: TestLockShardAddInstanceMapping (0.26s)
=== RUN   TestLockShardCloseInstance
--- PASS: TestLockShardCloseInstance (0.24s)
=== RUN   TestLockShardEmptyShard
--- PASS: TestLockShardEmptyShard (0.13s)
=== RUN   TestDestroyShardID
--- PASS: TestDestroyShardID (0.53s)
=== RUN   TestDestroyShardIDConcurrent
--- PASS: TestDestroyShardIDConcurrent (0.14s)
=== RUN   TestDestroyShardIDNumTenants
--- PASS: TestDestroyShardIDNumTenants (0.39s)
=== RUN   TestDestroyShardIDTenantAddRemove
--- PASS: TestDestroyShardIDTenantAddRemove (0.23s)
=== RUN   TestDestroyShardIDOnRestoreFailure
--- PASS: TestDestroyShardIDOnRestoreFailure (0.55s)
=== RUN   TestTransferShardAPI
--- PASS: TestTransferShardAPI (1.84s)
=== RUN   TestTransferShardAPICreateIndexes
--- PASS: TestTransferShardAPICreateIndexes (11.09s)
=== RUN   TestTransferShardAPIWithDropIndexes
--- PASS: TestTransferShardAPIWithDropIndexes (5.72s)
=== RUN   TestTransferShardAPIWithCancel
--- PASS: TestTransferShardAPIWithCancel (6.02s)
=== RUN   TestTransferShardAPIWithCleanup
--- PASS: TestTransferShardAPIWithCleanup (51.19s)
=== RUN   TestRestoreShardAPI
--- PASS: TestRestoreShardAPI (0.73s)
=== RUN   TestRestoreShardNumShards
--- PASS: TestRestoreShardNumShards (0.25s)
=== RUN   TestRestoreShardInvalidLocation
--- PASS: TestRestoreShardInvalidLocation (0.08s)
=== RUN   TestRestoreShardReplicaRepair
--- PASS: TestRestoreShardReplicaRepair (11.73s)
=== RUN   TestShardDoCleanupAPI
--- PASS: TestShardDoCleanupAPI (1.17s)
=== RUN   TestShardDoCleanupAPI2
--- PASS: TestShardDoCleanupAPI2 (15.62s)
=== RUN   TestDiag
--- PASS: TestDiag (0.47s)
=== RUN   TestDumpLog
--- PASS: TestDumpLog (0.09s)
=== RUN   TestExtrasN1
=== RUN   TestExtrasN2
=== RUN   TestExtrasN3
=== RUN   TestGMRecovery
--- PASS: TestGMRecovery (8.45s)
=== RUN   TestIteratorSimple
--- PASS: TestIteratorSimple (4.94s)
=== RUN   TestIteratorSeek
--- PASS: TestIteratorSeek (6.09s)
=== RUN   TestPlasmaIteratorSeekFirst
--- PASS: TestPlasmaIteratorSeekFirst (0.53s)
=== RUN   TestPlasmaIteratorSwapin
--- PASS: TestPlasmaIteratorSwapin (5.37s)
=== RUN   TestIteratorSetEnd
--- PASS: TestIteratorSetEnd (0.74s)
=== RUN   TestIterHiItm
--- PASS: TestIterHiItm (1.76s)
=== RUN   TestIterDeleteSplitMerge
--- PASS: TestIterDeleteSplitMerge (0.04s)
=== RUN   TestKeySamplingSingle
--- PASS: TestKeySamplingSingle (0.09s)
=== RUN   TestKeySamplingAll
--- PASS: TestKeySamplingAll (0.13s)
=== RUN   TestKeySamplingEmpty
--- PASS: TestKeySamplingEmpty (0.05s)
=== RUN   TestKeySamplingExceed
--- PASS: TestKeySamplingExceed (0.10s)
=== RUN   TestLogOperation
--- PASS: TestLogOperation (60.34s)
=== RUN   TestLogLargeSize
--- PASS: TestLogLargeSize (0.18s)
=== RUN   TestLogTrim
--- PASS: TestLogTrim (60.50s)
=== RUN   TestLogSuperblockCorruption
--- PASS: TestLogSuperblockCorruption (59.99s)
=== RUN   TestLogTrimHolePunch
--- PASS: TestLogTrimHolePunch (49.82s)
=== RUN   TestLogMissingAndTruncatedSegments
--- PASS: TestLogMissingAndTruncatedSegments (0.07s)
=== RUN   TestLogReadBeyondMaxFileIndex
--- PASS: TestLogReadBeyondMaxFileIndex (2.49s)
=== RUN   TestLogReadEOFWithMMap
--- PASS: TestLogReadEOFWithMMap (0.00s)
=== RUN   TestShardLSSCleaning
--- PASS: TestShardLSSCleaning (0.22s)
=== RUN   TestShardLSSCleaningDeleteInstance
--- PASS: TestShardLSSCleaningDeleteInstance (0.17s)
=== RUN   TestShardLSSCleaningCorruptInstance
--- PASS: TestShardLSSCleaningCorruptInstance (0.16s)
=== RUN   TestPlasmaLSSCleaner
--- PASS: TestPlasmaLSSCleaner (218.48s)
=== RUN   TestLSSBasic
--- PASS: TestLSSBasic (0.10s)
=== RUN   TestLSSConcurrent
--- PASS: TestLSSConcurrent (0.93s)
=== RUN   TestLSSCleaner
--- PASS: TestLSSCleaner (13.02s)
=== RUN   TestLSSSuperBlock
--- PASS: TestLSSSuperBlock (1.15s)
=== RUN   TestLSSLargeSinglePayload
--- PASS: TestLSSLargeSinglePayload (0.84s)
=== RUN   TestLSSUnstableEnvironment
--- PASS: TestLSSUnstableEnvironment (10.23s)
=== RUN   TestLSSSmallFlushBuffer
--- PASS: TestLSSSmallFlushBuffer (0.01s)
=== RUN   TestLSSTrimFlushBufferGC
--- PASS: TestLSSTrimFlushBufferGC (1.55s)
=== RUN   TestLSSTrimFlushBufferNoIO
--- PASS: TestLSSTrimFlushBufferNoIO (30.01s)
=== RUN   TestLSSTrimFlushBufferWithIO
--- PASS: TestLSSTrimFlushBufferWithIO (39.48s)
=== RUN   TestLSSExtendFlushBufferWithIO
--- PASS: TestLSSExtendFlushBufferWithIO (30.02s)
=== RUN   TestLSSCtxTrimFlushBuffer
--- PASS: TestLSSCtxTrimFlushBuffer (3.71s)
=== RUN   TestLSSNegativeGetFlushBufferMemory
--- PASS: TestLSSNegativeGetFlushBufferMemory (0.01s)
=== RUN   TestLSSNegativeGetFlushBufferMemoryII
--- PASS: TestLSSNegativeGetFlushBufferMemoryII (0.01s)
=== RUN   TestMem
Plasma: Adaptive memory quota tuning (decrementing): RSS:931815424, freePercent:88.80839150172162, currentQuota=1099511627776, newQuota=1073741824, netGrowth=0, percent=99Plasma: Adaptive memory quota tuning (incrementing): RSS:931446784, freePercent: 88.80886818325546, currentQuota=0, newQuota=10995116277--- PASS: TestMem (15.02s)
=== RUN   TestCpu
--- PASS: TestCpu (14.61s)
=== RUN   TestTopTen20
--- PASS: TestTopTen20 (0.81s)
=== RUN   TestTopTen5
--- PASS: TestTopTen5 (0.23s)
=== RUN   TestMVCCSimple
--- PASS: TestMVCCSimple (0.18s)
=== RUN   TestMVCCLookup
--- PASS: TestMVCCLookup (0.13s)
=== RUN   TestMVCCIteratorRefresh
--- PASS: TestMVCCIteratorRefresh (4.83s)
=== RUN   TestMVCCIteratorRefreshEveryRow
--- PASS: TestMVCCIteratorRefreshEveryRow (0.78s)
=== RUN   TestMVCCGarbageCollection
--- PASS: TestMVCCGarbageCollection (0.11s)
=== RUN   TestMVCCRecoveryPoint
--- PASS: TestMVCCRecoveryPoint (1.79s)
=== RUN   TestMVCCRollbackMergeSibling
--- PASS: TestMVCCRollbackMergeSibling (0.06s)
=== RUN   TestMVCCRollbackCompact
--- PASS: TestMVCCRollbackCompact (0.08s)
=== RUN   TestMVCCRollbackSplit
--- PASS: TestMVCCRollbackSplit (0.05s)
=== RUN   TestMVCCRollbackItemsNotInSnapshot
--- PASS: TestMVCCRollbackItemsNotInSnapshot (0.16s)
=== RUN   TestMVCCRecoveryPointRollbackedSnapshot
--- PASS: TestMVCCRecoveryPointRollbackedSnapshot (1.05s)
=== RUN   TestMVCCRollbackBetweenRecoveryPoint
--- PASS: TestMVCCRollbackBetweenRecoveryPoint (0.84s)
=== RUN   TestMVCCRecoveryPointCrash
--- PASS: TestMVCCRecoveryPointCrash (0.10s)
=== RUN   TestMVCCIntervalGC
--- PASS: TestMVCCIntervalGC (0.20s)
=== RUN   TestMVCCItemsCount
--- PASS: TestMVCCItemsCount (0.33s)
=== RUN   TestLargeItems
--- PASS: TestLargeItems (108.71s)
=== RUN   TestTooLargeKey
--- PASS: TestTooLargeKey (3.46s)
=== RUN   TestMVCCItemUpdateSize
--- PASS: TestMVCCItemUpdateSize (0.26s)
=== RUN   TestEvictionStats
--- PASS: TestEvictionStats (0.45s)
=== RUN   TestReaderCacheStats
--- PASS: TestReaderCacheStats (1.15s)
=== RUN   TestInvalidSnapshot
--- PASS: TestInvalidSnapshot (0.88s)
=== RUN   TestEmptyKeyInsert
--- PASS: TestEmptyKeyInsert (0.04s)
=== RUN   TestMVCCRecoveryPointError
--- PASS: TestMVCCRecoveryPointError (0.04s)
=== RUN   TestMVCCReaderPurgeSequential
--- PASS: TestMVCCReaderPurgeSequential (0.21s)
=== RUN   TestMVCCReaderNoPurge
--- PASS: TestMVCCReaderNoPurge (0.19s)
=== RUN   TestMVCCReaderPurgeAfterUpdate
--- PASS: TestMVCCReaderPurgeAfterUpdate (0.21s)
=== RUN   TestMVCCReaderPurgeAfterRollback
--- PASS: TestMVCCReaderPurgeAfterRollback (0.24s)
=== RUN   TestMVCCReaderPurgeSimple
--- PASS: TestMVCCReaderPurgeSimple (0.08s)
=== RUN   TestMVCCReaderPurgeRandom
--- PASS: TestMVCCReaderPurgeRandom (0.20s)
=== RUN   TestMVCCReaderPurgePageFlag
--- PASS: TestMVCCReaderPurgePageFlag (0.11s)
=== RUN   TestMVCCPurgeRatioWithRollback
--- PASS: TestMVCCPurgeRatioWithRollback (15.92s)
=== RUN   TestComputeItemsCountMVCCWithRollbackI
--- PASS: TestComputeItemsCountMVCCWithRollbackI (0.11s)
=== RUN   TestComputeItemsCountMVCCWithRollbackII
--- PASS: TestComputeItemsCountMVCCWithRollbackII (0.22s)
=== RUN   TestComputeItemsCountMVCCWithRollbackIII
--- PASS: TestComputeItemsCountMVCCWithRollbackIII (0.09s)
=== RUN   TestComputeItemsCountMVCCWithRollbackIV
--- PASS: TestComputeItemsCountMVCCWithRollbackIV (0.09s)
=== RUN   TestMVCCPurgedRecordsWithCompactFullMarshalAndCascadedEmptyPagesMerge
--- PASS: TestMVCCPurgedRecordsWithCompactFullMarshalAndCascadedEmptyPagesMerge (1.74s)
=== RUN   TestMaxDeltaChainLenWithCascadedEmptyPagesMerge
--- PASS: TestMaxDeltaChainLenWithCascadedEmptyPagesMerge (1.49s)
=== RUN   TestAutoHoleCleaner
--- PASS: TestAutoHoleCleaner (33.00s)
=== RUN   TestAutoHoleCleaner5Indexes
--- PASS: TestAutoHoleCleaner5Indexes (194.42s)
=== RUN   TestIteratorReportedHoleRegionBoundary
--- PASS: TestIteratorReportedHoleRegionBoundary (0.16s)
=== RUN   TestFullRangeHoleScans
--- PASS: TestFullRangeHoleScans (0.33s)
=== RUN   TestOverlappingRangeHoleScans
--- PASS: TestOverlappingRangeHoleScans (0.33s)
=== RUN   TestMVCCIteratorSMRRefreshOnHoleScan
--- PASS: TestMVCCIteratorSMRRefreshOnHoleScan (7.52s)
=== RUN   TestAutoHoleCleanerWithRecovery
--- PASS: TestAutoHoleCleanerWithRecovery (2.93s)
=== RUN   TestPageMergeCorrectness2
--- PASS: TestPageMergeCorrectness2 (0.00s)
=== RUN   TestPageMergeCorrectness
--- PASS: TestPageMergeCorrectness (0.01s)
=== RUN   TestPageMarshalFull
--- PASS: TestPageMarshalFull (0.01s)
=== RUN   TestPageMergeMarshal
--- PASS: TestPageMergeMarshal (0.00s)
=== RUN   TestPageOperations
--- PASS: TestPageOperations (0.03s)
=== RUN   TestPageIterator
--- PASS: TestPageIterator (0.00s)
=== RUN   TestPageMarshal
--- PASS: TestPageMarshal (0.02s)
=== RUN   TestPageMergeCorrectness3
--- PASS: TestPageMergeCorrectness3 (0.00s)
=== RUN   TestPageHasDataRecords
--- PASS: TestPageHasDataRecords (0.00s)
=== RUN   TestPlasmaPageVisitor
--- PASS: TestPlasmaPageVisitor (4.52s)
=== RUN   TestPageRingVisitor
--- PASS: TestPageRingVisitor (4.35s)
=== RUN   TestPauseVisitorOnLowMemory
--- PASS: TestPauseVisitorOnLowMemory (1.12s)
=== RUN   TestCheckpointRecovery
--- PASS: TestCheckpointRecovery (7.77s)
=== RUN   TestPageCorruption
--- PASS: TestPageCorruption (0.79s)
=== RUN   TestCheckPointRecoveryFollowCleaning
--- PASS: TestCheckPointRecoveryFollowCleaning (0.11s)
=== RUN   TestFragmentationWithZeroItems
--- PASS: TestFragmentationWithZeroItems (1.13s)
=== RUN   TestEvictOnPersist
--- PASS: TestEvictOnPersist (0.16s)
=== RUN   TestPlasmaSimple
--- PASS: TestPlasmaSimple (13.60s)
=== RUN   TestPlasmaCompression
--- PASS: TestPlasmaCompression (0.03s)
=== RUN   TestPlasmaCompressionWrong
--- PASS: TestPlasmaCompressionWrong (0.03s)
=== RUN   TestPlasmaInMemCompression
--- PASS: TestPlasmaInMemCompression (0.04s)
=== RUN   TestPlasmaInMemCompressionZstd
--- PASS: TestPlasmaInMemCompressionZstd (0.03s)
=== RUN   TestPlasmaInMemCompressionWrong
--- PASS: TestPlasmaInMemCompressionWrong (0.04s)
=== RUN   TestSpoiledConfig
--- PASS: TestSpoiledConfig (0.05s)
=== RUN   TestPlasmaErrorFile
--- PASS: TestPlasmaErrorFile (0.04s)
=== RUN   TestPlasmaPersistor
--- PASS: TestPlasmaPersistor (9.83s)
=== RUN   TestPlasmaEvictionLSSDataSize
--- PASS: TestPlasmaEvictionLSSDataSize (0.03s)
=== RUN   TestPlasmaEviction
--- PASS: TestPlasmaEviction (29.98s)
=== RUN   TestConcurrDelOps
--- PASS: TestConcurrDelOps (69.83s)
=== RUN   TestPlasmaDataSize
--- PASS: TestPlasmaDataSize (0.05s)
=== RUN   TestLargeBasePage
--- PASS: TestLargeBasePage (59.55s)
=== RUN   TestLargeValue
--- PASS: TestLargeValue (98.08s)
=== RUN   TestPlasmaTooLargeKey
--- PASS: TestPlasmaTooLargeKey (3.33s)
=== RUN   TestEvictAfterMerge
--- PASS: TestEvictAfterMerge (0.12s)
=== RUN   TestEvictDirty
--- PASS: TestEvictDirty (0.16s)
=== RUN   TestEvictUnderQuota
--- PASS: TestEvictUnderQuota (60.12s)
=== RUN   TestEvictSetting
--- PASS: TestEvictSetting (1.19s)
=== RUN   TestBasePageAfterCompaction
--- PASS: TestBasePageAfterCompaction (0.15s)
=== RUN   TestSwapout
--- PASS: TestSwapout (0.05s)
=== RUN   TestSwapoutSplitBasePage
--- PASS: TestSwapoutSplitBasePage (0.04s)
=== RUN   TestCompactFullMarshal
--- PASS: TestCompactFullMarshal (0.09s)
=== RUN   TestPageStats
--- PASS: TestPageStats (1.96s)
=== RUN   TestPageStatsTinyIndex
--- PASS: TestPageStatsTinyIndex (0.16s)
=== RUN   TestPageStatsTinyIndexOnRecovery
--- PASS: TestPageStatsTinyIndexOnRecovery (0.12s)
=== RUN   TestPageStatsTinyIndexOnSplitAndMerge
--- PASS: TestPageStatsTinyIndexOnSplitAndMerge (0.05s)
=== RUN   TestPageCompress
--- PASS: TestPageCompress (0.05s)
=== RUN   TestPageCompressSwapin
--- PASS: TestPageCompressSwapin (0.09s)
=== RUN   TestPageCompressStats
--- PASS: TestPageCompressStats (0.65s)
=== RUN   TestPageDecompressStats
--- PASS: TestPageDecompressStats (0.06s)
=== RUN   TestSharedDedicatedDataSize
--- PASS: TestSharedDedicatedDataSize (3.58s)
=== RUN   TestLastRpSns
--- PASS: TestLastRpSns (0.09s)
=== RUN   TestPageCompressState
--- PASS: TestPageCompressState (0.05s)
=== RUN   TestPageCompressDuringBurst
--- PASS: TestPageCompressDuringBurst (0.05s)
=== RUN   TestPageDontDecompressDuringScan
--- PASS: TestPageDontDecompressDuringScan (0.11s)
=== RUN   TestPageDecompressAndCompressSwapin
--- PASS: TestPageDecompressAndCompressSwapin (2.08s)
=== RUN   TestPageCompressibleStat
--- PASS: TestPageCompressibleStat (0.49s)
=== RUN   TestPageCompressibleStatRecovery
--- PASS: TestPageCompressibleStatRecovery (0.23s)
=== RUN   TestPageCompressBeforeEvictPercent
--- PASS: TestPageCompressBeforeEvictPercent (0.69s)
=== RUN   TestPageCompressDecompressAfterDisable
--- PASS: TestPageCompressDecompressAfterDisable (0.68s)
=== RUN   TestPageKeepSwapinChainCompressed
--- PASS: TestPageKeepSwapinChainCompressed (0.05s)
=== RUN   TestEnforceSameCompressionAlgo
--- PASS: TestEnforceSameCompressionAlgo (0.28s)
=== RUN   TestPageCompressChangeAlgo
--- PASS: TestPageCompressChangeAlgo (0.52s)
=== RUN   TestWrittenDataSz
--- PASS: TestWrittenDataSz (3.08s)
=== RUN   TestWrittenDataSzAfterRecoveryCleaning
--- PASS: TestWrittenDataSzAfterRecoveryCleaning (3.64s)
=== RUN   TestWrittenHdrSz
--- PASS: TestWrittenHdrSz (2.94s)
=== RUN   TestPersistConfigUpgrade
--- PASS: TestPersistConfigUpgrade (0.01s)
=== RUN   TestLSSSegmentSize
--- PASS: TestLSSSegmentSize (0.19s)
=== RUN   TestPlasmaFlushBufferSzCfg
--- PASS: TestPlasmaFlushBufferSzCfg (0.17s)
=== RUN   TestCompactionCountwithCompactFullMarshal
--- PASS: TestCompactionCountwithCompactFullMarshal (0.12s)
=== RUN   TestCompactionCountwithCompactFullMarshalSMO
--- PASS: TestCompactionCountwithCompactFullMarshalSMO (0.04s)
=== RUN   TestPageHasDataRecordsOnCompactFullMarshal
--- PASS: TestPageHasDataRecordsOnCompactFullMarshal (0.11s)
=== RUN   TestPauseReaderOnLowMemory
--- PASS: TestPauseReaderOnLowMemory (1.05s)
=== RUN   TestRecoveryCleanerFragRatio
--- PASS: TestRecoveryCleanerFragRatio (220.47s)
=== RUN   TestRecoveryCleanerRelocation
--- PASS: TestRecoveryCleanerRelocation (221.42s)
=== RUN   TestRecoveryCleanerDataSize
--- PASS: TestRecoveryCleanerDataSize (220.24s)
=== RUN   TestRecoveryCleanerDeleteInstance
--- PASS: TestRecoveryCleanerDeleteInstance (434.95s)
=== RUN   TestRecoveryCleanerRecoveryPoint
--- PASS: TestRecoveryCleanerRecoveryPoint (27.05s)
=== RUN   TestRecoveryCleanerCorruptInstance
--- PASS: TestRecoveryCleanerCorruptInstance (0.18s)
=== RUN   TestRecoveryCleanerAhead
--- PASS: TestRecoveryCleanerAhead (4.21s)
=== RUN   TestRecoveryCleanerAheadAfterRecovery
--- PASS: TestRecoveryCleanerAheadAfterRecovery (2.23s)
=== RUN   TestCleaningUncommittedData
--- PASS: TestCleaningUncommittedData (0.04s)
=== RUN   TestPlasmaRecoverySimple
--- PASS: TestPlasmaRecoverySimple (0.05s)
=== RUN   TestPlasmaRecovery
--- PASS: TestPlasmaRecovery (24.47s)
=== RUN   TestShardRecoveryShared
--- PASS: TestShardRecoveryShared (10.33s)
=== RUN   TestShardRecoveryRecoveryLogAhead
--- PASS: TestShardRecoveryRecoveryLogAhead (31.99s)
=== RUN   TestShardRecoveryDataLogAhead
--- PASS: TestShardRecoveryDataLogAhead (21.29s)
=== RUN   TestShardRecoveryDestroyBlksInDataLog
--- PASS: TestShardRecoveryDestroyBlksInDataLog (9.66s)
=== RUN   TestShardRecoveryDestroyBlksInRecoveryLog
--- PASS: TestShardRecoveryDestroyBlksInRecoveryLog (9.87s)
=== RUN   TestShardRecoveryDestroyBlksInBothLog
--- PASS: TestShardRecoveryDestroyBlksInBothLog (9.43s)
=== RUN   TestShardRecoveryRecoveryLogCorruption
--- PASS: TestShardRecoveryRecoveryLogCorruption (9.22s)
=== RUN   TestShardRecoveryDataLogCorruption
--- PASS: TestShardRecoveryDataLogCorruption (10.00s)
=== RUN   TestShardRecoverySharedNoRP
--- PASS: TestShardRecoverySharedNoRP (10.11s)
=== RUN   TestShardRecoveryNotEnoughMem
--- PASS: TestShardRecoveryNotEnoughMem (32.90s)
=== RUN   TestShardRecoveryCleanup
--- PASS: TestShardRecoveryCleanup (0.47s)
=== RUN   TestShardRecoveryRebuildSharedLog
--- PASS: TestShardRecoveryRebuildSharedLog (1.33s)
=== RUN   TestShardRecoveryUpgradeWithCheckpoint
--- PASS: TestShardRecoveryUpgradeWithCheckpoint (0.50s)
=== RUN   TestShardRecoveryUpgradeWithLogReplay
--- PASS: TestShardRecoveryUpgradeWithLogReplay (0.46s)
=== RUN   TestShardRecoveryRebuildAfterError
--- PASS: TestShardRecoveryRebuildAfterError (1.28s)
=== RUN   TestShardRecoveryRebuildAfterConcurrentDelete
--- PASS: TestShardRecoveryRebuildAfterConcurrentDelete (1.65s)
=== RUN   TestShardRecoveryAfterDeleteInstance
--- PASS: TestShardRecoveryAfterDeleteInstance (0.15s)
=== RUN   TestShardRecoveryDestroyShard
--- PASS: TestShardRecoveryDestroyShard (0.26s)
=== RUN   TestHeaderRepair
--- PASS: TestHeaderRepair (0.08s)
=== RUN   TestCheckpointWithWriter
--- PASS: TestCheckpointWithWriter (3.28s)
=== RUN   TestPlasmaRecoveryWithRepairFullReplay
--- PASS: TestPlasmaRecoveryWithRepairFullReplay (27.03s)
=== RUN   TestPlasmaRecoveryWithInsertRepairCheckpoint
--- PASS: TestPlasmaRecoveryWithInsertRepairCheckpoint (45.73s)
=== RUN   TestPlasmaRecoveryWithDeleteRepairCheckpoint
--- PASS: TestPlasmaRecoveryWithDeleteRepairCheckpoint (12.99s)
=== RUN   TestShardRecoverySharedFullReplayOnError
--- PASS: TestShardRecoverySharedFullReplayOnError (11.92s)
=== RUN   TestShardRecoveryDedicatedFullReplayOnError
--- PASS: TestShardRecoveryDedicatedFullReplayOnError (12.10s)
=== RUN   TestShardRecoverySharedFullReplayOnErrorWithRepair
--- PASS: TestShardRecoverySharedFullReplayOnErrorWithRepair (13.92s)
=== RUN   TestGlobalWorkContextForRecovery
--- PASS: TestGlobalWorkContextForRecovery (0.33s)
=== RUN   TestShardRecoveryPartialMetadata
--- PASS: TestShardRecoveryPartialMetadata (0.13s)
=== RUN   TestShardRecoveryPartialLSSMetadata
--- PASS: TestShardRecoveryPartialLSSMetadata (0.09s)
=== RUN   TestShardRecoveryPartialSkiplog
--- PASS: TestShardRecoveryPartialSkiplog (0.14s)
=== RUN   TestShardRecoveryTempSkiplog
--- PASS: TestShardRecoveryTempSkiplog (0.24s)
=== RUN   TestSkipLogSimple
--- PASS: TestSkipLogSimple (0.00s)
=== RUN   TestSkipLogLoadStore
--- PASS: TestSkipLogLoadStore (0.03s)
=== RUN   TestShardMetadata
--- PASS: TestShardMetadata (0.08s)
=== RUN   TestPlasmaId
--- PASS: TestPlasmaId (0.05s)
=== RUN   TestShardPersistence
--- PASS: TestShardPersistence (0.23s)
=== RUN   TestShardDestroy
--- PASS: TestShardDestroy (0.07s)
=== RUN   TestShardClose
--- PASS: TestShardClose (5.04s)
=== RUN   TestShardMgrRecovery
--- PASS: TestShardMgrRecovery (0.12s)
=== RUN   TestShardDeadData
--- PASS: TestShardDeadData (0.21s)
=== RUN   TestShardConfigUpdate
--- PASS: TestShardConfigUpdate (0.09s)
=== RUN   TestShardWriteAmp
--- PASS: TestShardWriteAmp (10.14s)
=== RUN   TestShardStats
--- PASS: TestShardStats (0.20s)
=== RUN   TestShardMultipleWriters
--- PASS: TestShardMultipleWriters (0.14s)
=== RUN   TestShardDestroyMultiple
--- PASS: TestShardDestroyMultiple (0.16s)
=== RUN   TestShardBackupCorrupted
--- PASS: TestShardBackupCorrupted (0.17s)
=== RUN   TestShardBackupCorruptedShare
--- PASS: TestShardBackupCorruptedShare (0.07s)
=== RUN   TestShardCorruption
--- PASS: TestShardCorruption (0.11s)
=== RUN   TestShardCorruptionAddInstance
--- PASS: TestShardCorruptionAddInstance (0.24s)
=== RUN   TestShardCreateError
--- PASS: TestShardCreateError (0.23s)
=== RUN   TestShardNumInsts
--- PASS: TestShardNumInsts (1.71s)
=== RUN   TestShardInstanceGroup
--- PASS: TestShardInstanceGroup (0.11s)
=== RUN   TestShardLeak
--- PASS: TestShardLeak (1.83s)
=== RUN   TestShardMemLeak
--- PASS: TestShardMemLeak (0.79s)
=== RUN   TestShardFind
--- PASS: TestShardFind (0.26s)
=== RUN   TestShardFileOpenDescCount
--- PASS: TestShardFileOpenDescCount (55.39s)
=== RUN   TestShardShutdownSharedLSS
--- PASS: TestShardShutdownSharedLSS (6.90s)
=== RUN   TestShardUUIDChange
--- PASS: TestShardUUIDChange (0.09s)
=== RUN   TestShardUUIDStable
--- PASS: TestShardUUIDStable (0.07s)
=== RUN   TestShardMetadataPersistenceOnPanic
--- PASS: TestShardMetadataPersistenceOnPanic (0.18s)
=== RUN   TestSMRSimple
--- PASS: TestSMRSimple (1.11s)
=== RUN   TestSMRConcurrent
--- PASS: TestSMRConcurrent (50.95s)
=== RUN   TestSMRComplex
--- PASS: TestSMRComplex (116.14s)
=== RUN   TestDGMWithCASConflicts
--- PASS: TestDGMWithCASConflicts (29.60s)
=== RUN   TestMaxSMRPendingMem
--- PASS: TestMaxSMRPendingMem (0.04s)
=== RUN   TestStatsLogger
--- PASS: TestStatsLogger (20.35s)
=== RUN   TestStatsSamplePercentile
--- PASS: TestStatsSamplePercentile (0.04s)
=== RUN   TestPlasmaSwapper
--- PASS: TestPlasmaSwapper (21.91s)
=== RUN   TestPlasmaAutoSwapper
--- PASS: TestPlasmaAutoSwapper (82.15s)
=== RUN   TestSwapperAddInstance
--- PASS: TestSwapperAddInstance (4.37s)
=== RUN   TestSwapperRemoveInstance
--- PASS: TestSwapperRemoveInstance (5.20s)
=== RUN   TestSwapperJoinContext
--- PASS: TestSwapperJoinContext (4.96s)
=== RUN   TestSwapperSplitContext
--- PASS: TestSwapperSplitContext (5.02s)
=== RUN   TestSwapperGlobalClock
--- PASS: TestSwapperGlobalClock (29.76s)
=== RUN   TestSwapperConflict
--- PASS: TestSwapperConflict (2.85s)
=== RUN   TestSwapperRemoveInstanceWait
--- PASS: TestSwapperRemoveInstanceWait (3.44s)
=== RUN   TestSwapperStats
--- PASS: TestSwapperStats (0.97s)
=== RUN   TestSwapperSweepInterval
--- PASS: TestSwapperSweepInterval (0.46s)
=== RUN   TestSweepCompress
--- PASS: TestSweepCompress (0.08s)
=== RUN   TestTenantShardAssignment
--- PASS: TestTenantShardAssignment (3.05s)
=== RUN   TestTenantShardAssignmentServerless
--- PASS: TestTenantShardAssignmentServerless (13.93s)
=== RUN   TestTenantShardAssignmentDedicated
--- PASS: TestTenantShardAssignmentDedicated (1.64s)
=== RUN   TestTenantShardAssignmentDedicatedMainBackIndexes
--- PASS: TestTenantShardAssignmentDedicatedMainBackIndexes (0.12s)
=== RUN   TestTenantShardRecovery
--- PASS: TestTenantShardRecovery (2.80s)
=== RUN   TestTenantMemUsed
--- PASS: TestTenantMemUsed (2.87s)
=== RUN   TestTenantSwitchController
--- PASS: TestTenantSwitchController (0.10s)
=== RUN   TestTenantControllerSwapperIncremental
--- PASS: TestTenantControllerSwapperIncremental (0.89s)
=== RUN   TestTenantControllerSwapperInitial
--- PASS: TestTenantControllerSwapperInitial (0.89s)
=== RUN   TestTenantControllerSwapperPeriodic
--- PASS: TestTenantControllerSwapperPeriodic (2.50s)
=== RUN   TestTenantAssignMandatoryQuota
--- PASS: TestTenantAssignMandatoryQuota (0.46s)
=== RUN   TestTenantMutationQuota
--- PASS: TestTenantMutationQuota (0.13s)
=== RUN   TestTenantInitialBuildQuota
--- PASS: TestTenantInitialBuildQuota (0.08s)
=== RUN   TestTenantInitialBuildNonDGM
--- PASS: TestTenantInitialBuildNonDGM (1.88s)
=== RUN   TestTenantInitialBuildDGM
--- PASS: TestTenantInitialBuildDGM (1.89s)
=== RUN   TestTenantInitialBuildZeroResident
--- PASS: TestTenantInitialBuildZeroResident (1.87s)
=== RUN   TestTenantIncrementalBuildDGM
--- PASS: TestTenantIncrementalBuildDGM (3.02s)
=== RUN   TestTenantInitialBuildTwoTenants
--- PASS: TestTenantInitialBuildTwoTenants (2.96s)
=== RUN   TestTenantInitialBuildTwoControllers
--- PASS: TestTenantInitialBuildTwoControllers (2.98s)
=== RUN   TestTenantIncrementalBuildTwoIndexes
--- PASS: TestTenantIncrementalBuildTwoIndexes (0.37s)
=== RUN   TestTenantIncrementalBuildConcurrent
--- PASS: TestTenantIncrementalBuildConcurrent (4.96s)
=== RUN   TestTenantDecrementGlobalQuota
--- PASS: TestTenantDecrementGlobalQuota (2.28s)
=== RUN   TestTenantInitialBuildNotEnoughQuota
--- PASS: TestTenantInitialBuildNotEnoughQuota (3.23s)
=== RUN   TestTenantRecoveryResidentRatioHeaderReplay
--- PASS: TestTenantRecoveryResidentRatioHeaderReplay (0.16s)
=== RUN   TestTenantRecoveryResidentRatioDataReplay
--- PASS: TestTenantRecoveryResidentRatioDataReplay (0.20s)
=== RUN   TestTenantRecoveryController
--- PASS: TestTenantRecoveryController (1.57s)
=== RUN   TestTenantRecoveryQuotaWithLastCheckpoint
--- PASS: TestTenantRecoveryQuotaWithLastCheckpoint (0.78s)
=== RUN   TestTenantRecoveryQuotaZeroResidentWithLastCheckpoint
--- PASS: TestTenantRecoveryQuotaZeroResidentWithLastCheckpoint (3.14s)
=== RUN   TestTenantRecoveryQuotaWithFormula
--- PASS: TestTenantRecoveryQuotaWithFormula (3.13s)
=== RUN   TestTenantRecoveryQuotaWithDataReplay
--- PASS: TestTenantRecoveryQuotaWithDataReplay (7.15s)
=== RUN   TestTenantRecoveryEvictionNoCheckpoint
--- PASS: TestTenantRecoveryEvictionNoCheckpoint (15.15s)
=== RUN   TestTenantRecoveryEvictionHeaderReplay
--- PASS: TestTenantRecoveryEvictionHeaderReplay (9.25s)
=== RUN   TestTenantRecoveryEvictionDataReplaySequential
--- PASS: TestTenantRecoveryEvictionDataReplaySequential (9.98s)
=== RUN   TestTenantRecoveryEvictionDataReplayInterleaved
--- PASS: TestTenantRecoveryEvictionDataReplayInterleaved (10.01s)
=== RUN   TestTenantRecoveryEvictionDataReplayNoCheckpoint
--- PASS: TestTenantRecoveryEvictionDataReplayNoCheckpoint (9.99s)
=== RUN   TestTenantRecoveryEvictionDataReplaySingle
--- PASS: TestTenantRecoveryEvictionDataReplaySingle (4.41s)
=== RUN   TestTenantRecoveryLastCheckpoint
--- PASS: TestTenantRecoveryLastCheckpoint (8.00s)
=== RUN   TestTenantRecoveryRequestQuota
--- PASS: TestTenantRecoveryRequestQuota (2.57s)
=== RUN   TestTenantWorkingSetPageHits
--- PASS: TestTenantWorkingSetPageHits (0.04s)
=== RUN   TestTenantWorkingSetConcurrent
--- PASS: TestTenantWorkingSetConcurrent (0.27s)
=== RUN   TestTenantDiscretionaryQuotaNoScan
--- PASS: TestTenantDiscretionaryQuotaNoScan (3.17s)
=== RUN   TestTenantDiscretionaryQuotaScan
--- PASS: TestTenantDiscretionaryQuotaScan (9.26s)
=== RUN   TestTenantAssignDiscretionaryQuota
--- PASS: TestTenantAssignDiscretionaryQuota (24.50s)
=== RUN   TestTenantMinimumQuota
--- PASS: TestTenantMinimumQuota (7.39s)
=== RUN   TestTenantWorkingSetIdle
--- PASS: TestTenantWorkingSetIdle (0.03s)
=== RUN   TestTenantWorkingSetSinceIdle
--- PASS: TestTenantWorkingSetSinceIdle (9.59s)
=== RUN   TestTenantWorkingSetIdleTime
--- PASS: TestTenantWorkingSetIdleTime (1.03s)
=== RUN   TestTenantAssignIdleQuota
--- PASS: TestTenantAssignIdleQuota (8.01s)
=== RUN   TestTenantActivateIdleTenant
--- PASS: TestTenantActivateIdleTenant (22.66s)
=== RUN   TestTenantStatsInStr
--- PASS: TestTenantStatsInStr (0.03s)
=== RUN   TestTenantMandatoryQuotaLowResident
--- PASS: TestTenantMandatoryQuotaLowResident (51.89s)
=== RUN   TestTenantNumShards
--- PASS: TestTenantNumShards (1.89s)
=== RUN   TestTenantNumShardsMixedConfig
--- PASS: TestTenantNumShardsMixedConfig (2.76s)
=== RUN   TestWriteRenameFileWithSync
--- PASS: TestWriteRenameFileWithSync (0.00s)
=== RUN   TestSCtx
--- PASS: TestSCtx (17.80s)
=== RUN   TestWCtxGeneric
--- PASS: TestWCtxGeneric (22.29s)
=== RUN   TestWCtxWriter
--- PASS: TestWCtxWriter (21.62s)
=== RUN   TestSCtxTrimWithReader
--- PASS: TestSCtxTrimWithReader (0.06s)
=== RUN   TestSCtxTrimWithWriter
--- PASS: TestSCtxTrimWithWriter (0.05s)
=== RUN   TestSCtxTrimEmpty
--- PASS: TestSCtxTrimEmpty (0.04s)
=== RUN   TestWCtxTrimWithReader
--- PASS: TestWCtxTrimWithReader (0.05s)
=== RUN   TestWCtxTrimWithWriter
--- PASS: TestWCtxTrimWithWriter (0.06s)
--- PASS: TestExtrasN1 (0.00s)
--- PASS: TestExtrasN3 (0.00s)
--- PASS: TestExtrasN2 (0.00s)
PASS
ok  	github.com/couchbase/plasma	3969.288s
/home/buildbot/bin/dotest: line 243: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
/home/buildbot/bin/dotest: line 246: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
/home/buildbot/bin/dotest: line 249: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 262: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 267: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Serverless tests

Starting server: attempt 1

Serverless tests

/home/buildbot/bin/dotest: line 287: /home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.7/go/bin/go: No such file or directory
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_serverless_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_2/indexer_serverless_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9114 after 1 ms: Connection refused

Integration tests

git submodule init; git submodule update --init --force
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '5dd338995c16ac2f5b187729e549b28862060732'
Submodule path 'lib/capellaAPI': checked out 'eaee55e75f8d02f4cb435d5d89c8062db614b894'
Submodule path 'magma_loader/DocLoader': checked out 'bd442d704d3aa88e6cbb06b7bc7dfc68dd675c66'
echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[  0%] Built target ns_ui_build_prepare
[  0%] Building Go Modules target ns_minify_js using Go 1.18.7
CMake Error at /opt/build/tlm/cmake/Modules/go-modbuild.cmake:140 (MESSAGE):
  Error creating temporary mod file
  /opt/build/ns_server/build/deps/gomodules/minify_js/minify_js-go.mod:


../ns_server/build/deps/gomodules/minify_js/CMakeFiles/ns_minify_js.dir/build.make:70: recipe for target '../ns_server/build/deps/gomodules/minify_js/CMakeFiles/ns_minify_js' failed
make[4]: *** [../ns_server/build/deps/gomodules/minify_js/CMakeFiles/ns_minify_js] Error 1
CMakeFiles/Makefile2:16915: recipe for target '../ns_server/build/deps/gomodules/minify_js/CMakeFiles/ns_minify_js.dir/all' failed
make[3]: *** [../ns_server/build/deps/gomodules/minify_js/CMakeFiles/ns_minify_js.dir/all] Error 2
Makefile:165: recipe for target 'all' failed
make[2]: *** [all] Error 2
GNUmakefile:19: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-11-25 17:22:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:09 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:09 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:09 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-11-25 17:22:12 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-11-25 17:22:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.5.0-3184-rel-EE-enterprise
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:32 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-11-25 17:22:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:37 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-11-25 17:22:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-11-25 17:22:38 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:38 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-11-25 17:22:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 6343634944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:22:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:22:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 6164701184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:22:39 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-11-25 17:22:39 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:40 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:22:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '26', 'memoryTotal': 15466930176, 'memoryFree': 6021787648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:22:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:22:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 5942980608, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:22:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-11-25 17:22:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:22:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:22:42 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-11-25 17:22:42 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-11-25 17:22:42 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-11-25 17:22:43 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:22:48 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-11-25 17:22:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-11-25 17:22:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-11-25 17:22:48 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-11-25 17:22:49 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:22:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:22:50 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 44.98552794684673, 'mem_free': 13659242496, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.54820203781128
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:22:53 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-11-25 17:22:54 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-11-25 17:22:59 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-11-25 17:22:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:22:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:22:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:00 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-11-25 17:23:00 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:23:02 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-11-25 17:23:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:02 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-11-25 17:23:02 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 156.620927ms
2022-11-25 17:23:32 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'cc15012d-248d-49f1-8c15-136ded479bea', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '156.620927ms', 'executionTime': '156.370423ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-11-25 17:23:32 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 791.271432ms
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.684762ms
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-11-25 17:23:33 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.149485ms
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-11-25 17:23:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 36.227766ms
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.834544100325989, 'mem_free': 13636108288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:35 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:39 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-11-25 17:23:39 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-11-25 17:23:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-11-25 17:23:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-11-25 17:23:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 91.902s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.5.0-3184-rel-EE-enterprise
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-11-25 17:23:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.62295870749155, 'mem_free': 13611753472, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-11-25 17:23:46 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-11-25 17:23:46 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-11-25 17:23:47 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13611753472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-11-25 17:23:48 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-11-25 17:23:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13605748736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13606268928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:23:49 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13604937728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-11-25 17:23:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-11-25 17:23:50 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-11-25 17:23:51 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-11-25 17:23:56 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-11-25 17:23:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-11-25 17:23:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-11-25 17:23:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-11-25 17:23:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:23:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:23:58 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:23:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:23:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:23:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.56267995103052, 'mem_free': 13586874368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.65233302116394
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:24:02 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-11-25 17:24:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-11-25 17:24:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-11-25 17:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:09 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-11-25 17:24:09 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-11-25 17:24:12 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-11-25 17:24:12 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:12 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-11-25 17:24:12 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 75.447887ms
2022-11-25 17:24:42 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '07bf168e-8628-4175-ae66-ffe231447141', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '75.447887ms', 'executionTime': '75.364538ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-11-25 17:24:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 719.404591ms
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.871832ms
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-11-25 17:24:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeea8e78b59913b4777bb3ae156b82ff431job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-11-25 17:24:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeea8e78b59913b4777bb3ae156b82ff431job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-11-25 17:24:44 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 77.720869ms
2022-11-25 17:24:44 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeea8e78b59913b4777bb3ae156b82ff431job_title) USING GSI
2022-11-25 17:24:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeea8e78b59913b4777bb3ae156b82ff431job_title) USING GSI
2022-11-25 17:24:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeea8e78b59913b4777bb3ae156b82ff431job_title%29+USING+GSI
2022-11-25 17:24:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 28.151124ms
2022-11-25 17:24:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea8e78b59913b4777bb3ae156b82ff431job_title'
2022-11-25 17:24:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8e78b59913b4777bb3ae156b82ff431job_title%27
2022-11-25 17:24:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.8607ms
2022-11-25 17:24:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea8e78b59913b4777bb3ae156b82ff431job_title'
2022-11-25 17:24:47 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8e78b59913b4777bb3ae156b82ff431job_title%27
2022-11-25 17:24:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.06593ms
2022-11-25 17:24:48 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-11-25 17:24:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-11-25 17:24:49 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.120823ms
2022-11-25 17:24:49 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '4f19a1a2-8080-4cba-8326-d7ac69f633e6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea8e78b59913b4777bb3ae156b82ff431job_title', 'index_id': '67f7cdc5d700dee9', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.120823ms', 'executionTime': '2.935288ms', 'resultCount': 1, 'resultSize': 746, 'serviceLoad': 6}}
2022-11-25 17:24:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:49 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-11-25 17:24:49 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 137.648241ms
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:50 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea8e78b59913b4777bb3ae156b82ff431job_title'
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8e78b59913b4777bb3ae156b82ff431job_title%27
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.271793ms
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeea8e78b59913b4777bb3ae156b82ff431job_title ON default USING GSI
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeea8e78b59913b4777bb3ae156b82ff431job_title+ON+default+USING+GSI
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 46.49191ms
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeea8e78b59913b4777bb3ae156b82ff431job_title'
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea8e78b59913b4777bb3ae156b82ff431job_title%27
2022-11-25 17:24:52 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.195997ms
2022-11-25 17:24:52 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '9ba8bc48-52be-4cd1-970b-928c108b971b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.195997ms', 'executionTime': '6.048336ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.77804ms
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.382752ms
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.34534705110563, 'mem_free': 13488492544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-11-25 17:24:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-11-25 17:24:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-11-25 17:24:58 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-11-25 17:24:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-11-25 17:24:59 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-11-25 17:24:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-11-25 17:25:00 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 78.908s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_2'}
[2022-11-25 17:25:00,152] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:00,153] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:00,436] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:00,472] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:25:00,555] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:25:00,555] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-11-25 17:25:00,556] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:25:01,182] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:25:01,210] - [task:165] INFO -  {'uptime': '168', 'memoryTotal': 15466930176, 'memoryFree': 13532913664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:25:01,237] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:25:01,237] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:25:01,237] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:25:01,269] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:25:01,307] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:25:01,307] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:25:01,334] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:25:01,335] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:25:01,335] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:25:01,335] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:25:01,383] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:25:01,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:01,387] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:01,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:01,681] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:25:01,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:25:01,763] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:25:01,796] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:01,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:01,857] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:25:01,992] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:25:02,022] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13495885824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:25:02,052] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:25:02,084] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:25:02,084] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:25:02,139] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:25:02,142] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:02,142] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:02,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:02,457] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:25:02,535] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:25:02,536] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:25:02,567] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:02,595] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:02,627] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:25:02,760] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:25:02,787] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13495992320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:25:02,815] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:25:02,845] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:25:02,846] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:25:02,903] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:25:02,907] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:02,908] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:03,186] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:03,187] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:25:03,262] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:25:03,263] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:25:03,294] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:03,321] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:03,351] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:25:03,468] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:25:03,496] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13533683712, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:25:03,523] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:25:03,553] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:25:03,554] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:25:03,606] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:25:03,609] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:03,610] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:03,900] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:03,901] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:25:03,973] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:25:03,974] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:25:04,003] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:04,029] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:25:04,057] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:25:04,153] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:25:04,760] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:25:09,761] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:25:09,848] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:25:09,853] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:09,859] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:10,144] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:10,145] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:25:10,217] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:25:10,219] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:25:10,219] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:25:11,285] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:25:11,341] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:25:11,341] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:25:12,351] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:25:12,678] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:25:12,988] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:25:13,046] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:25:13,102] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:13,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:13,397] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:13,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:13,402] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:13,657] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:13,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:13,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:13,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:13,961] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:13,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:14,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:18,479] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:25:18,480] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.71846131753754, 'mem_free': 13519847424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:25:18,480] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:25:18,480] - [basetestcase:466] INFO - Time to execute basesetup : 18.33011484146118
[2022-11-25 17:25:18,536] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:25:18,537] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:25:18,597] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:25:18,598] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:25:18,657] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:25:18,658] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:25:18,717] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:25:18,718] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:25:18,776] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:25:18,865] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:25:18,866] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:25:18,866] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:25:23,879] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:25:23,883] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:25:23,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:25:24,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:25:25,179] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:25:25,372] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:25:27,918] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:25:27,939] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-11-25 17:25:28,606] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:25:28,607] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:25:28,607] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:25:58,635] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:25:58,665] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:25:58,693] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:25:58,766] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.328825ms
[2022-11-25 17:25:58,766] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5b417404-fa95-4562-ab0f-cd6f3225e81a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.328825ms', 'executionTime': '70.243801ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:25:58,766] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:25:58,794] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:25:58,822] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:25:59,587] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 762.032258ms
[2022-11-25 17:25:59,587] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:25:59,662] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:25:59,712] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:25:59,721] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.047146ms
[2022-11-25 17:25:59,938] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:25:59,976] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:25:59,990] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:25:59,990] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:26:00,001] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:26:00,093] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:00,893] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:26:00,921] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:26:00,999] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 75.20685ms
[2022-11-25 17:26:00,999] - [base_gsi:288] INFO - BUILD INDEX on default(employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title) USING GSI
[2022-11-25 17:26:02,033] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title) USING GSI
[2022-11-25 17:26:02,062] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%29+USING+GSI
[2022-11-25 17:26:02,085] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.073144ms
[2022-11-25 17:26:03,118] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title'
[2022-11-25 17:26:03,146] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%27
[2022-11-25 17:26:03,156] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.76778ms
[2022-11-25 17:26:04,188] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title'
[2022-11-25 17:26:04,221] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%27
[2022-11-25 17:26:04,231] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.919304ms
[2022-11-25 17:26:04,717] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-11-25 17:26:04,897] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:26:05,536] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:26:06,235] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:06,268] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:26:06,295] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-11-25 17:26:06,300] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.163906ms
[2022-11-25 17:26:06,300] - [task:3246] INFO - {'requestID': '8dd23571-d814-4038-89c6-f3108198a79d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title', 'index_id': 'a85475c0fcdb5ae', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.163906ms', 'executionTime': '3.091281ms', 'resultCount': 1, 'resultSize': 745, 'serviceLoad': 6}}
[2022-11-25 17:26:06,300] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:06,300] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:06,301] - [base_gsi:566] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:26:06,301] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:26:06,302] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-11-25 17:26:06,302] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:26:06,302] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:26:06,302] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:26:06,302] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:26:07,302] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:07,334] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:26:07,359] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-11-25 17:26:07,493] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 123.914427ms
[2022-11-25 17:26:07,493] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:26:07,494] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:26:08,293] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:08,293] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:26:09,326] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title'
[2022-11-25 17:26:09,353] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%27
[2022-11-25 17:26:09,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.572981ms
[2022-11-25 17:26:09,392] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title ON default USING GSI
[2022-11-25 17:26:09,419] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title+ON+default+USING+GSI
[2022-11-25 17:26:09,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.453708ms
[2022-11-25 17:26:09,501] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title'
[2022-11-25 17:26:09,528] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeff500d39e5148ddaa3f5a82b9b8ca26job_title%27
[2022-11-25 17:26:09,536] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.517514ms
[2022-11-25 17:26:09,537] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6fa84a27-9ff6-4b63-bb83-db0fc4d395bb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.517514ms', 'executionTime': '6.439362ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:26:09,647] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:09,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:09,652] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:10,036] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:10,096] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:26:10,096] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:26:10,164] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:26:10,224] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:10,228] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:10,228] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:10,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:10,692] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:26:10,692] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:26:10,760] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:26:10,820] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:10,821] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:26:10,821] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:26:10,849] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:26:10,879] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:26:10,889] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.896137ms
[2022-11-25 17:26:10,918] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:26:10,945] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:26:10,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.619962ms
[2022-11-25 17:26:11,072] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:26:11,073] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.51233469785027, 'mem_free': 13411946496, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:26:11,073] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:26:11,077] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:11,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:11,465] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:11,470] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:11,470] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:12,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:12,067] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:12,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:12,706] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:12,715] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:12,715] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:13,366] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:17,302] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:26:17,574] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:26:17,953] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:26:17,983] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:26:17,983] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:26:18,041] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:26:18,095] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:26:18,149] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:26:18,151] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:26:18,230] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:26:18,231] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:26:18,262] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:18,393] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:26:18,394] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:18,422] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:26:18,449] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:26:18,450] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:18,477] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:26:18,503] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:26:18,503] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:18,529] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:26:18,555] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:26:18,556] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:18,585] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:26:18,585] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:26:18,585] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:26:18,585] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 78.489s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_3'}
[2022-11-25 17:26:18,673] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:18,673] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:19,016] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:19,047] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:26:19,127] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:26:19,127] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-11-25 17:26:19,128] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:26:19,705] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:26:19,735] - [task:165] INFO -  {'uptime': '246', 'memoryTotal': 15466930176, 'memoryFree': 13465993216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:26:19,764] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:26:19,764] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:26:19,764] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:26:19,801] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:26:19,836] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:26:19,836] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:26:19,866] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:26:19,867] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:26:19,868] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:26:19,868] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:26:19,919] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:26:19,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:19,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:20,281] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:20,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:26:20,370] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:26:20,371] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:26:20,403] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:20,432] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:20,464] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:26:20,596] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:26:20,625] - [task:165] INFO -  {'uptime': '246', 'memoryTotal': 15466930176, 'memoryFree': 13467226112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:26:20,654] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:26:20,686] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:26:20,686] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:26:20,742] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:26:20,747] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:20,747] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:21,118] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:21,119] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:26:21,205] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:26:21,206] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:26:21,237] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:21,265] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:21,296] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:26:21,422] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:26:21,451] - [task:165] INFO -  {'uptime': '242', 'memoryTotal': 15466930176, 'memoryFree': 13467602944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:26:21,480] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:26:21,510] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:26:21,510] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:26:21,565] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:26:21,568] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:21,568] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:21,903] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:21,904] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:26:21,983] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:26:21,984] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:26:22,018] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:22,044] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:22,076] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:26:22,196] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:26:22,224] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13466886144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:26:22,250] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:26:22,280] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:26:22,281] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:26:22,333] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:26:22,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:22,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:22,654] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:22,655] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:26:22,738] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:26:22,739] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:26:22,772] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:22,797] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:26:22,825] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:26:22,915] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:26:23,606] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:26:28,611] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:26:28,696] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:26:28,705] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:28,706] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:29,034] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:29,036] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:26:29,111] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:26:29,112] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:26:29,112] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:26:30,047] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:26:30,103] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:26:30,104] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:26:31,041] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:26:31,371] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:26:31,673] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:26:31,675] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:26:31,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:31,743] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:32,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:32,120] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:32,121] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:32,457] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:32,462] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:32,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:32,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:32,926] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:32,927] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:33,526] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:37,816] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:26:37,817] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.04129015467542, 'mem_free': 13475209216, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:26:37,817] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:26:37,817] - [basetestcase:466] INFO - Time to execute basesetup : 19.146304607391357
[2022-11-25 17:26:37,871] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:26:37,871] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:26:37,924] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:26:37,925] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:26:37,978] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:26:37,978] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:26:38,035] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:26:38,036] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:26:38,090] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:38,147] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:26:38,147] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:26:38,148] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:26:43,159] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:26:43,162] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:26:43,162] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:26:43,520] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:26:44,582] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:26:44,757] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:26:47,284] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:26:47,301] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-11-25 17:26:48,160] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:26:48,160] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:26:48,160] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:27:18,190] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:27:18,222] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:27:18,250] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:27:18,327] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.756949ms
[2022-11-25 17:27:18,327] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a977e0f1-fe13-4c47-9a07-70b1494d72d0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '74.756949ms', 'executionTime': '74.681875ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:27:18,327] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:27:18,356] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:27:18,385] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:27:19,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 751.479487ms
[2022-11-25 17:27:19,139] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:27:19,211] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:27:19,245] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:27:19,254] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.582645ms
[2022-11-25 17:27:19,495] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:27:19,551] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:27:19,569] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:27:19,569] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:27:19,584] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:27:19,649] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:27:20,418] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee1013cd4099a74605b05dca56a2aab800job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:27:20,446] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee1013cd4099a74605b05dca56a2aab800job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:27:20,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.58955ms
[2022-11-25 17:27:20,504] - [base_gsi:288] INFO - BUILD INDEX on default(employee1013cd4099a74605b05dca56a2aab800job_title) USING GSI
[2022-11-25 17:27:21,537] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee1013cd4099a74605b05dca56a2aab800job_title) USING GSI
[2022-11-25 17:27:21,566] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee1013cd4099a74605b05dca56a2aab800job_title%29+USING+GSI
[2022-11-25 17:27:21,592] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.691999ms
[2022-11-25 17:27:22,626] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1013cd4099a74605b05dca56a2aab800job_title'
[2022-11-25 17:27:22,655] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1013cd4099a74605b05dca56a2aab800job_title%27
[2022-11-25 17:27:22,665] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.965709ms
[2022-11-25 17:27:23,697] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1013cd4099a74605b05dca56a2aab800job_title'
[2022-11-25 17:27:23,725] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1013cd4099a74605b05dca56a2aab800job_title%27
[2022-11-25 17:27:23,734] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.34282ms
[2022-11-25 17:27:24,238] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-11-25 17:27:24,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:27:24,881] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:27:25,736] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:25,771] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:27:25,797] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-11-25 17:27:25,802] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.033634ms
[2022-11-25 17:27:25,802] - [task:3246] INFO - {'requestID': 'd24ecbe2-3b0d-4474-a100-e0d5b1378eb5', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee1013cd4099a74605b05dca56a2aab800job_title', 'index_id': '7012c67dec3d870c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.033634ms', 'executionTime': '2.934009ms', 'resultCount': 1, 'resultSize': 746, 'serviceLoad': 6}}
[2022-11-25 17:27:25,802] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:25,803] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:25,803] - [base_gsi:566] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:27:25,804] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:27:25,804] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-11-25 17:27:25,804] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:27:25,804] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:27:25,805] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:27:25,805] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:27:26,804] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:26,837] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:27:26,865] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-11-25 17:27:27,041] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 164.858517ms
[2022-11-25 17:27:27,041] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:27:27,041] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:27:27,825] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:27,826] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:27:28,859] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1013cd4099a74605b05dca56a2aab800job_title'
[2022-11-25 17:27:28,886] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1013cd4099a74605b05dca56a2aab800job_title%27
[2022-11-25 17:27:28,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.27661ms
[2022-11-25 17:27:28,923] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee1013cd4099a74605b05dca56a2aab800job_title ON default USING GSI
[2022-11-25 17:27:28,949] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee1013cd4099a74605b05dca56a2aab800job_title+ON+default+USING+GSI
[2022-11-25 17:27:28,990] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.112174ms
[2022-11-25 17:27:29,026] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1013cd4099a74605b05dca56a2aab800job_title'
[2022-11-25 17:27:29,053] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1013cd4099a74605b05dca56a2aab800job_title%27
[2022-11-25 17:27:29,063] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.291091ms
[2022-11-25 17:27:29,063] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f55d538f-9869-46d3-aad2-04a25b09d498', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.291091ms', 'executionTime': '7.216298ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:27:29,182] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:27:29,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:29,186] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:29,625] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:29,684] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:27:29,685] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:27:29,770] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:27:29,829] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:27:29,832] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:29,832] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:30,299] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:30,364] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:27:30,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:27:30,441] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:27:30,510] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:27:30,511] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:27:30,511] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:27:30,541] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:27:30,570] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:27:30,578] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.995963ms
[2022-11-25 17:27:30,608] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:27:30,637] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:27:30,703] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.655787ms
[2022-11-25 17:27:30,781] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:27:30,781] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.149609804593308, 'mem_free': 13375418368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:27:30,781] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:27:30,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:30,786] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:31,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:31,251] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:31,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:31,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:31,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:31,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:32,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:32,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:32,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:33,158] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:38,274] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:27:38,422] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:27:39,645] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:27:39,674] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:27:39,675] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:27:39,737] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:27:39,793] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:27:39,846] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:27:39,848] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:27:39,938] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:27:39,939] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:27:39,966] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,097] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:27:40,097] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,124] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:27:40,150] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:27:40,151] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,177] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:27:40,203] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:27:40,203] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,232] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:27:40,258] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:27:40,258] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,285] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:27:40,286] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:27:40,286] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:27:40,287] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 81.669s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_4'}
[2022-11-25 17:27:40,392] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:40,393] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:40,846] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:40,880] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:27:40,962] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:27:40,962] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-11-25 17:27:40,963] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:27:41,419] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:27:41,446] - [task:165] INFO -  {'uptime': '328', 'memoryTotal': 15466930176, 'memoryFree': 13431115776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:27:41,474] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:27:41,474] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:27:41,475] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:27:41,514] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:27:41,548] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:27:41,548] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:27:41,578] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:27:41,579] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:27:41,579] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:27:41,580] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:27:41,629] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:27:41,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:41,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:42,006] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:42,007] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:27:42,092] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:27:42,093] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:27:42,124] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:42,149] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:42,179] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:27:42,311] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:27:42,338] - [task:165] INFO -  {'uptime': '326', 'memoryTotal': 15466930176, 'memoryFree': 13430607872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:27:42,366] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:27:42,398] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:27:42,398] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:27:42,449] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:27:42,452] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:42,452] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:42,832] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:42,833] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:27:42,919] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:27:42,920] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:27:42,949] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:42,975] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:43,005] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:27:43,122] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:27:43,148] - [task:165] INFO -  {'uptime': '327', 'memoryTotal': 15466930176, 'memoryFree': 13431758848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:27:43,174] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:27:43,203] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:27:43,203] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:27:43,254] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:27:43,260] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:43,260] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:43,648] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:43,649] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:27:43,733] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:27:43,734] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:27:43,764] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:43,789] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:43,819] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:27:43,935] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:27:43,963] - [task:165] INFO -  {'uptime': '326', 'memoryTotal': 15466930176, 'memoryFree': 13431390208, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:27:43,989] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:27:44,017] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:27:44,018] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:27:44,069] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:27:44,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:44,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:44,448] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:44,449] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:27:44,538] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:27:44,539] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:27:44,569] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:44,594] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:27:44,622] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:27:44,722] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:27:45,238] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:27:50,243] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:27:50,336] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:27:50,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:50,341] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:50,786] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:50,787] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:27:50,878] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:27:50,879] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:27:50,880] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:27:51,889] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:27:51,946] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:27:51,947] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:27:52,921] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:27:53,245] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:27:53,525] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:27:53,528] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:27:53,587] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:53,587] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:54,016] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:54,021] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:54,021] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:54,441] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:54,446] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:54,446] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:55,133] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:27:55,142] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:27:55,142] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:27:55,862] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:01,024] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:28:01,024] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 23.29339441509078, 'mem_free': 13430964224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:28:01,025] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:28:01,025] - [basetestcase:466] INFO - Time to execute basesetup : 20.634968280792236
[2022-11-25 17:28:01,083] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:28:01,084] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:28:01,144] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:28:01,144] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:28:01,206] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:28:01,207] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:28:01,269] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:28:01,270] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:28:01,329] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:01,400] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:28:01,402] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:28:01,403] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:28:06,415] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:28:06,419] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:06,420] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:06,827] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:07,959] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:28:08,157] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:28:10,576] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:28:10,599] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-11-25 17:28:11,314] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:11,314] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:28:11,314] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:28:41,342] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:28:41,375] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:28:41,404] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:28:41,484] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 77.78224ms
[2022-11-25 17:28:41,484] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '26b8669b-7318-4133-8855-2ed582a150db', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '77.78224ms', 'executionTime': '77.707172ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:28:41,484] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:28:41,514] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:28:41,543] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:28:42,314] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 768.05485ms
[2022-11-25 17:28:42,314] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:28:42,369] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:28:42,409] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:28:42,418] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.416346ms
[2022-11-25 17:28:42,653] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:42,693] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:28:42,708] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:28:42,708] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:28:42,720] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:28:42,820] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:43,607] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee1cb9507cb32744a6b6b98bb21916260ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:28:43,636] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee1cb9507cb32744a6b6b98bb21916260ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:28:43,706] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.780215ms
[2022-11-25 17:28:43,706] - [base_gsi:288] INFO - BUILD INDEX on default(employee1cb9507cb32744a6b6b98bb21916260ajob_title) USING GSI
[2022-11-25 17:28:44,740] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee1cb9507cb32744a6b6b98bb21916260ajob_title) USING GSI
[2022-11-25 17:28:44,769] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee1cb9507cb32744a6b6b98bb21916260ajob_title%29+USING+GSI
[2022-11-25 17:28:44,799] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.789359ms
[2022-11-25 17:28:45,833] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1cb9507cb32744a6b6b98bb21916260ajob_title'
[2022-11-25 17:28:45,862] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1cb9507cb32744a6b6b98bb21916260ajob_title%27
[2022-11-25 17:28:45,871] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.402786ms
[2022-11-25 17:28:46,904] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1cb9507cb32744a6b6b98bb21916260ajob_title'
[2022-11-25 17:28:46,934] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1cb9507cb32744a6b6b98bb21916260ajob_title%27
[2022-11-25 17:28:46,944] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.011388ms
[2022-11-25 17:28:47,474] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-11-25 17:28:47,669] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:28:48,178] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:28:48,388] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-11-25 17:28:48,388] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:28:48,586] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-11-25 17:28:48,586] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-11-25 17:28:48,587] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3993474434, 0, b'')
[2022-11-25 17:28:48,947] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:48,978] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:28:49,007] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-11-25 17:28:49,012] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.504718ms
[2022-11-25 17:28:49,012] - [task:3246] INFO - {'requestID': '5ed82360-22b0-41cc-ab06-6a30647752ca', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee1cb9507cb32744a6b6b98bb21916260ajob_title', 'index_id': 'affd51b2e66f522b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.504718ms', 'executionTime': '2.430981ms', 'resultCount': 1, 'resultSize': 746, 'serviceLoad': 6}}
[2022-11-25 17:28:49,012] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:49,013] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:49,013] - [base_gsi:566] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:28:49,013] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:28:49,014] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-11-25 17:28:49,014] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:28:49,014] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:28:49,014] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:28:49,015] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:28:50,014] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:50,047] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:28:50,077] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-11-25 17:28:50,277] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 191.28699ms
[2022-11-25 17:28:50,277] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:28:50,278] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:28:51,049] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:51,050] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:28:52,084] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1cb9507cb32744a6b6b98bb21916260ajob_title'
[2022-11-25 17:28:52,115] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1cb9507cb32744a6b6b98bb21916260ajob_title%27
[2022-11-25 17:28:52,126] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.703062ms
[2022-11-25 17:28:52,156] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee1cb9507cb32744a6b6b98bb21916260ajob_title ON default USING GSI
[2022-11-25 17:28:52,186] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee1cb9507cb32744a6b6b98bb21916260ajob_title+ON+default+USING+GSI
[2022-11-25 17:28:52,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 363.147909ms
[2022-11-25 17:28:52,587] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1cb9507cb32744a6b6b98bb21916260ajob_title'
[2022-11-25 17:28:52,619] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1cb9507cb32744a6b6b98bb21916260ajob_title%27
[2022-11-25 17:28:52,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.47301ms
[2022-11-25 17:28:52,628] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b12add0f-6e76-46cb-ac7a-9b4279554ce2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.47301ms', 'executionTime': '6.39045ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:28:52,759] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:52,762] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:52,762] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:53,710] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:53,777] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:28:53,778] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:28:53,863] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:28:53,932] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:53,935] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:53,935] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:54,750] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:54,826] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:28:54,827] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:28:54,917] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:28:54,981] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:28:54,982] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:28:54,982] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:28:55,012] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:28:55,041] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:28:55,049] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.448401ms
[2022-11-25 17:28:55,080] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:28:55,109] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:28:55,504] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 392.270832ms
[2022-11-25 17:28:55,573] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:28:55,573] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.30848273777892, 'mem_free': 13287985152, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:28:55,574] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:28:55,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:55,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:56,377] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:56,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:56,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:57,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:57,174] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:57,174] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:58,022] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:28:58,034] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:28:58,034] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:28:58,884] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:04,534] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:29:04,688] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:29:05,456] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:29:05,486] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:29:05,486] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:29:05,544] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:29:05,599] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:29:05,652] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:29:05,654] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:29:05,735] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:29:05,736] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:29:05,763] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:05,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:29:05,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:05,943] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:29:05,970] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:29:05,970] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:05,997] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:29:06,024] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:29:06,025] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:06,054] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:29:06,080] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:29:06,081] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:06,107] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:29:06,107] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:29:06,107] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:29:06,108] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 85.770s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_5'}
[2022-11-25 17:29:06,214] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:06,214] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:06,665] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:06,700] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:29:06,781] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:29:06,781] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-11-25 17:29:06,782] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:29:07,242] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:29:07,271] - [task:165] INFO -  {'uptime': '414', 'memoryTotal': 15466930176, 'memoryFree': 13320847360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:29:07,298] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:29:07,299] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:29:07,299] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:29:07,332] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:29:07,369] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:29:07,369] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:29:07,400] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:29:07,401] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:29:07,401] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:29:07,401] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:29:07,451] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:29:07,455] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:07,455] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:07,938] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:07,940] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:29:08,044] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:29:08,045] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:29:08,077] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:08,108] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:08,139] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:29:08,271] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:29:08,301] - [task:165] INFO -  {'uptime': '411', 'memoryTotal': 15466930176, 'memoryFree': 13404565504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:29:08,329] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:29:08,359] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:29:08,359] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:29:08,415] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:29:08,418] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:08,419] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:08,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:08,910] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:29:09,018] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:29:09,020] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:29:09,052] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:09,080] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:09,110] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:29:09,237] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:29:09,267] - [task:165] INFO -  {'uptime': '412', 'memoryTotal': 15466930176, 'memoryFree': 13404532736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:29:09,295] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:29:09,325] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:29:09,325] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:29:09,378] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:29:09,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:09,384] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:09,880] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:09,881] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:29:09,987] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:29:09,988] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:29:10,019] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:10,046] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:10,076] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:29:10,194] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:29:10,222] - [task:165] INFO -  {'uptime': '412', 'memoryTotal': 15466930176, 'memoryFree': 13404524544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:29:10,249] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:29:10,278] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:29:10,279] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:29:10,331] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:29:10,334] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:10,334] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:10,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:10,823] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:29:10,922] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:29:10,923] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:29:10,953] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:10,981] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:29:11,013] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:29:11,107] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:29:11,731] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:29:16,736] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:29:16,827] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:29:16,831] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:16,832] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:17,311] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:17,312] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:29:17,404] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:29:17,406] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:29:17,406] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:29:18,224] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:29:18,279] - [on_prem_rest_client:3058] INFO - 0.05 seconds to create bucket default
[2022-11-25 17:29:18,280] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:29:19,204] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:29:19,537] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:29:19,823] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:29:19,826] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:29:19,884] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:19,884] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:20,367] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:20,373] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:20,373] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:20,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:20,889] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:20,890] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:21,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:21,667] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:21,667] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:22,482] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:27,889] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:29:27,889] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.751918160358, 'mem_free': 13409009664, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:29:27,889] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:29:27,890] - [basetestcase:466] INFO - Time to execute basesetup : 21.678305625915527
[2022-11-25 17:29:27,942] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:29:27,943] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:29:27,999] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:29:27,999] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:29:28,057] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:29:28,057] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:29:28,113] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:29:28,113] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:29:28,167] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:29:28,232] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:29:28,232] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:29:28,233] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:29:33,244] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:29:33,247] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:29:33,248] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:29:33,752] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:29:34,889] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:29:35,085] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:29:37,450] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:29:37,470] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-11-25 17:29:38,563] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:29:38,564] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:29:38,564] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:30:08,591] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:30:08,623] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:30:08,652] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:30:08,729] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 74.252052ms
[2022-11-25 17:30:08,730] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '30c5b2eb-52c3-4b6c-8d20-24931ad8f3a6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '74.252052ms', 'executionTime': '74.170939ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:30:08,730] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:30:08,760] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:30:08,789] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:30:09,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 759.899198ms
[2022-11-25 17:30:09,552] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:30:09,609] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:30:09,651] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:30:09,660] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.220838ms
[2022-11-25 17:30:09,888] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:30:09,930] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:30:09,947] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:30:09,948] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:30:09,965] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:30:10,038] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:30:10,834] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee988708fef0d843a6950aeceb32cfa15ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:30:10,862] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee988708fef0d843a6950aeceb32cfa15ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:30:10,928] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.463262ms
[2022-11-25 17:30:10,928] - [base_gsi:288] INFO - BUILD INDEX on default(employee988708fef0d843a6950aeceb32cfa15ejob_title) USING GSI
[2022-11-25 17:30:11,961] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee988708fef0d843a6950aeceb32cfa15ejob_title) USING GSI
[2022-11-25 17:30:11,988] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee988708fef0d843a6950aeceb32cfa15ejob_title%29+USING+GSI
[2022-11-25 17:30:12,012] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.195003ms
[2022-11-25 17:30:13,050] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee988708fef0d843a6950aeceb32cfa15ejob_title'
[2022-11-25 17:30:13,078] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee988708fef0d843a6950aeceb32cfa15ejob_title%27
[2022-11-25 17:30:13,088] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.753069ms
[2022-11-25 17:30:14,120] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee988708fef0d843a6950aeceb32cfa15ejob_title'
[2022-11-25 17:30:14,148] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee988708fef0d843a6950aeceb32cfa15ejob_title%27
[2022-11-25 17:30:14,158] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.407624ms
[2022-11-25 17:30:14,685] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:30:14,880] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:30:18,378] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:30:19,163] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:19,197] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:30:19,226] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-11-25 17:30:19,231] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.046888ms
[2022-11-25 17:30:19,231] - [task:3246] INFO - {'requestID': 'b4ba2b5b-ac40-4f58-bf8d-090d2b709454', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee988708fef0d843a6950aeceb32cfa15ejob_title', 'index_id': 'e5e214fa8a833171', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.046888ms', 'executionTime': '2.968701ms', 'resultCount': 1, 'resultSize': 746, 'serviceLoad': 6}}
[2022-11-25 17:30:19,231] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:19,232] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:19,232] - [base_gsi:566] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:30:19,232] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:30:19,233] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-11-25 17:30:19,233] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:30:19,233] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:30:19,234] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:30:19,234] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:30:20,233] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:20,267] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:30:20,294] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-11-25 17:30:20,488] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 178.043774ms
[2022-11-25 17:30:20,488] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:30:20,489] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:30:22,030] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:22,031] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:30:23,063] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee988708fef0d843a6950aeceb32cfa15ejob_title'
[2022-11-25 17:30:23,095] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee988708fef0d843a6950aeceb32cfa15ejob_title%27
[2022-11-25 17:30:23,106] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.692428ms
[2022-11-25 17:30:23,134] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee988708fef0d843a6950aeceb32cfa15ejob_title ON default USING GSI
[2022-11-25 17:30:23,161] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee988708fef0d843a6950aeceb32cfa15ejob_title+ON+default+USING+GSI
[2022-11-25 17:30:23,203] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.049768ms
[2022-11-25 17:30:23,241] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee988708fef0d843a6950aeceb32cfa15ejob_title'
[2022-11-25 17:30:23,269] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee988708fef0d843a6950aeceb32cfa15ejob_title%27
[2022-11-25 17:30:23,277] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.836154ms
[2022-11-25 17:30:23,277] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ffc06a07-5456-4929-8b6a-8d557990dfda', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.836154ms', 'executionTime': '5.758514ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:30:23,395] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:30:23,398] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:23,398] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:23,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:24,049] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:30:24,049] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:30:24,144] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:30:24,201] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:30:24,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:24,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:24,795] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:24,861] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:30:24,862] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:30:24,961] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:30:25,019] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:30:25,019] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:30:25,020] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:30:25,047] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:30:25,075] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:30:25,083] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.565695ms
[2022-11-25 17:30:25,110] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:30:25,136] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:30:25,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.428565ms
[2022-11-25 17:30:25,274] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:30:25,275] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.62422212489957, 'mem_free': 13247746048, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:30:25,275] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:30:25,281] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:25,281] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:25,865] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:25,870] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:25,871] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:26,657] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:26,665] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:26,666] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:27,672] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:27,679] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:27,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:28,702] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:35,137] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:30:35,297] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:30:35,824] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:30:35,854] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:30:35,855] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:30:35,913] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:30:35,972] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:30:36,030] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:30:36,032] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:30:36,118] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:30:36,119] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:30:36,147] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:36,295] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:30:36,295] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:36,325] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:30:36,353] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:30:36,353] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:36,383] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:30:36,412] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:30:36,412] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:36,455] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:30:36,483] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:30:36,483] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:36,513] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:30:36,513] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:30:36,513] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:30:36,514] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 90.361s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_6'}
[2022-11-25 17:30:36,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:36,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:37,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:37,261] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:30:37,345] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:30:37,346] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-11-25 17:30:37,346] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:30:37,662] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:30:37,691] - [task:165] INFO -  {'uptime': '504', 'memoryTotal': 15466930176, 'memoryFree': 13316440064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:30:37,718] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:30:37,718] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:30:37,719] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:30:37,754] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:30:37,797] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:30:37,797] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:30:37,827] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:30:37,828] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:30:37,828] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:30:37,828] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:30:37,878] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:30:37,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:37,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:38,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:38,462] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:30:38,578] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:30:38,579] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:30:38,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:38,639] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:38,672] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:30:38,811] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:30:38,843] - [task:165] INFO -  {'uptime': '502', 'memoryTotal': 15466930176, 'memoryFree': 13313441792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:30:38,873] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:30:38,904] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:30:38,904] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:30:38,962] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:30:38,965] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:38,965] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:39,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:39,544] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:30:39,656] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:30:39,657] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:30:39,687] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:39,715] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:39,746] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:30:39,864] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:30:39,892] - [task:165] INFO -  {'uptime': '503', 'memoryTotal': 15466930176, 'memoryFree': 13312999424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:30:39,922] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:30:39,953] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:30:39,954] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:30:40,006] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:30:40,012] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:40,012] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:40,610] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:40,611] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:30:40,726] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:30:40,727] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:30:40,758] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:40,786] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:40,816] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:30:40,947] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:30:40,978] - [task:165] INFO -  {'uptime': '502', 'memoryTotal': 15466930176, 'memoryFree': 13313470464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:30:41,006] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:30:41,036] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:30:41,036] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:30:41,090] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:30:41,093] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:41,094] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:41,674] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:41,675] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:30:41,793] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:30:41,795] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:30:41,827] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:41,855] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:30:41,888] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:30:41,984] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:30:42,529] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:30:47,533] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:30:47,620] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:30:47,625] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:47,625] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:48,203] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:48,204] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:30:48,323] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:30:48,324] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:30:48,324] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:30:49,132] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:30:49,193] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:30:49,194] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:30:50,295] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:30:50,665] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:30:50,974] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:30:50,977] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:30:51,037] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:51,037] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:51,626] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:51,631] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:51,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:52,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:52,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:52,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:53,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:30:53,385] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:30:53,385] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:30:54,351] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:31:00,453] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:31:00,453] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.46302135853515, 'mem_free': 13314326528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:31:00,453] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:31:00,454] - [basetestcase:466] INFO - Time to execute basesetup : 23.822117567062378
[2022-11-25 17:31:00,508] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:31:00,509] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:31:00,571] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:31:00,572] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:31:00,630] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:31:00,630] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:31:00,688] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:31:00,689] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:31:00,749] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:31:00,820] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:31:00,820] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:31:00,821] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:31:05,835] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:31:05,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:31:05,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:31:06,406] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:31:07,598] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:31:07,777] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:31:09,916] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:31:09,938] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-11-25 17:31:11,507] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:31:11,507] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:31:11,508] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:31:41,532] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:31:41,563] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:31:41,591] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:31:41,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 72.711388ms
[2022-11-25 17:31:41,666] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c298a3ad-c2af-4f9a-96ef-bbaff2447d5a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '72.711388ms', 'executionTime': '72.637168ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:31:41,666] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:31:41,695] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:31:41,722] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:31:42,478] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 753.865778ms
[2022-11-25 17:31:42,478] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:31:42,551] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:31:42,590] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:31:42,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.422547ms
[2022-11-25 17:31:42,812] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:31:42,854] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:31:42,873] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:31:42,874] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:31:42,889] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:31:42,955] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:31:43,775] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5a7c81d7b07f42278e099005eb72a1c5job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:31:43,801] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee5a7c81d7b07f42278e099005eb72a1c5job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:31:43,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 72.87739ms
[2022-11-25 17:31:43,878] - [base_gsi:288] INFO - BUILD INDEX on default(employee5a7c81d7b07f42278e099005eb72a1c5job_title) USING GSI
[2022-11-25 17:31:44,918] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5a7c81d7b07f42278e099005eb72a1c5job_title) USING GSI
[2022-11-25 17:31:44,946] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee5a7c81d7b07f42278e099005eb72a1c5job_title%29+USING+GSI
[2022-11-25 17:31:44,971] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.587977ms
[2022-11-25 17:31:46,004] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5a7c81d7b07f42278e099005eb72a1c5job_title'
[2022-11-25 17:31:46,031] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5a7c81d7b07f42278e099005eb72a1c5job_title%27
[2022-11-25 17:31:46,042] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.486701ms
[2022-11-25 17:31:47,075] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5a7c81d7b07f42278e099005eb72a1c5job_title'
[2022-11-25 17:31:47,104] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5a7c81d7b07f42278e099005eb72a1c5job_title%27
[2022-11-25 17:31:47,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.308659ms
[2022-11-25 17:31:47,509] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-11-25 17:31:47,690] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:31:48,531] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:31:49,211] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-11-25 17:31:49,400] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:31:50,584] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:31:51,283] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:31:51,463] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:31:54,929] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:31:55,170] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:55,205] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:31:55,233] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-11-25 17:31:55,238] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.229002ms
[2022-11-25 17:31:55,238] - [task:3246] INFO - {'requestID': '81740f63-03dc-4297-9865-a52b885ef4ca', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5a7c81d7b07f42278e099005eb72a1c5job_title', 'index_id': 'e1e6cceaf8479d32', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.229002ms', 'executionTime': '3.159426ms', 'resultCount': 1, 'resultSize': 746, 'serviceLoad': 6}}
[2022-11-25 17:31:55,239] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:55,239] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:55,239] - [base_gsi:566] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:31:55,240] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:31:55,240] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-11-25 17:31:55,240] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:31:55,240] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:31:55,241] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:31:55,241] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-11-25 17:31:56,240] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:56,271] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-11-25 17:31:56,297] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-11-25 17:31:56,512] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 198.513616ms
[2022-11-25 17:31:56,512] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:31:56,513] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:31:58,076] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:58,077] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-11-25 17:31:59,108] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5a7c81d7b07f42278e099005eb72a1c5job_title'
[2022-11-25 17:31:59,135] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5a7c81d7b07f42278e099005eb72a1c5job_title%27
[2022-11-25 17:31:59,144] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.865837ms
[2022-11-25 17:31:59,173] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5a7c81d7b07f42278e099005eb72a1c5job_title ON default USING GSI
[2022-11-25 17:31:59,199] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee5a7c81d7b07f42278e099005eb72a1c5job_title+ON+default+USING+GSI
[2022-11-25 17:31:59,257] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.216638ms
[2022-11-25 17:31:59,293] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5a7c81d7b07f42278e099005eb72a1c5job_title'
[2022-11-25 17:31:59,321] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5a7c81d7b07f42278e099005eb72a1c5job_title%27
[2022-11-25 17:31:59,329] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.663258ms
[2022-11-25 17:31:59,329] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '598963f4-640e-4aa1-b61c-e67f23ac42dd', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.663258ms', 'executionTime': '5.544729ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:31:59,444] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:31:59,448] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:31:59,448] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:00,248] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:00,318] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:32:00,318] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:32:00,432] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:32:00,495] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:32:00,498] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:00,498] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:01,252] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:01,313] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:32:01,313] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:32:01,429] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:32:01,495] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:32:01,495] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:32:01,496] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:32:01,525] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:32:01,555] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:32:01,566] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.241666ms
[2022-11-25 17:32:01,597] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:32:01,627] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:32:01,678] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.606584ms
[2022-11-25 17:32:01,754] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:32:01,754] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 36.40046008626617, 'mem_free': 13120503808, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:32:01,754] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:32:01,759] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:01,759] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:02,493] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:02,499] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:02,499] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:03,476] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:03,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:03,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:04,698] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:04,707] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:04,707] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:05,884] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:13,070] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:32:13,217] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:32:13,702] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:32:13,730] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:32:13,730] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:32:13,789] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:32:13,844] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:32:13,896] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:32:13,897] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:32:13,978] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:32:13,979] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:32:14,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:14,136] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:32:14,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:14,163] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:32:14,189] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:32:14,189] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:14,216] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:32:14,241] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:32:14,242] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:14,270] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:32:14,297] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:32:14,298] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:14,326] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:32:14,327] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-11-25 17:32:14,327] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:32:14,328] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.767s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_7'}
[2022-11-25 17:32:14,492] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:14,493] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:15,163] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:15,198] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:32:15,282] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:32:15,283] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-11-25 17:32:15,283] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:32:15,521] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:32:15,551] - [task:165] INFO -  {'uptime': '602', 'memoryTotal': 15466930176, 'memoryFree': 13262278656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:32:15,580] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:32:15,580] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:32:15,580] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:32:15,612] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:32:15,653] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:32:15,654] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:32:15,685] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:32:15,686] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:32:15,686] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:32:15,686] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:32:15,738] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:32:15,742] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:15,742] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:16,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:16,414] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:32:16,541] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:32:16,542] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:32:16,574] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:16,602] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:16,634] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:32:16,763] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:32:16,792] - [task:165] INFO -  {'uptime': '602', 'memoryTotal': 15466930176, 'memoryFree': 13261107200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:32:16,820] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:32:16,851] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:32:16,852] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:32:16,907] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:32:16,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:16,910] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:17,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:17,576] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:32:17,705] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:32:17,707] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:32:17,738] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:17,767] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:17,799] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:32:17,927] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:32:17,956] - [task:165] INFO -  {'uptime': '598', 'memoryTotal': 15466930176, 'memoryFree': 13260283904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:32:17,984] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:32:18,015] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:32:18,015] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:32:18,071] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:32:18,077] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:18,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:18,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:18,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:32:18,903] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:32:18,905] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:32:18,936] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:18,964] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:18,994] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:32:19,119] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:32:19,150] - [task:165] INFO -  {'uptime': '598', 'memoryTotal': 15466930176, 'memoryFree': 13251723264, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:32:19,179] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:32:19,210] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:32:19,211] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:32:19,270] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:32:19,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:19,274] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:19,962] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:19,963] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:32:20,096] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:32:20,097] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:32:20,129] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:20,156] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:32:20,187] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:32:20,289] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:32:20,872] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:32:25,876] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:32:25,966] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:32:25,972] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:25,973] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:26,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:26,633] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:32:26,766] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:32:26,767] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:32:26,767] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:32:27,455] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:32:27,514] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:32:27,515] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:32:28,466] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:32:28,830] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:32:29,173] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:32:29,177] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-11-25 17:32:29,238] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:29,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:29,923] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:29,928] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:29,928] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:30,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:30,965] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:30,965] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:32,105] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:32,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:32,115] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:33,273] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:40,223] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:32:40,224] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.82521029529944, 'mem_free': 13272227840, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:32:40,224] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:32:40,224] - [basetestcase:466] INFO - Time to execute basesetup : 25.734283685684204
[2022-11-25 17:32:40,279] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:32:40,279] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:32:40,336] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:32:40,336] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:32:40,390] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:32:40,391] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:32:40,449] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:32:40,449] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:32:40,546] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:32:40,621] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:32:40,622] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:32:40,622] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:32:45,632] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:32:45,636] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:32:45,636] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:32:46,328] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:32:47,444] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:32:47,782] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:32:50,738] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:32:50,824] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:32:50,825] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:32:50,825] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:33:20,855] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:33:20,886] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:33:20,914] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:33:20,987] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.901104ms
[2022-11-25 17:33:20,987] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '491283a5-dd06-4e82-be4b-3b007c60fd23', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.901104ms', 'executionTime': '70.829041ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:33:20,988] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:33:21,015] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:33:21,043] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:33:21,762] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 717.127904ms
[2022-11-25 17:33:21,763] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:33:21,822] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:33:21,867] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:33:21,876] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.974112ms
[2022-11-25 17:33:22,082] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:33:22,120] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:33:22,146] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:33:22,152] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:33:22,169] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:33:22,170] - [base_gsi:332] INFO - []
[2022-11-25 17:33:23,046] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee015df01c6b434b3196588f049489ab5fjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:33:23,073] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee015df01c6b434b3196588f049489ab5fjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:33:23,161] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 85.546478ms
[2022-11-25 17:33:23,197] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee015df01c6b434b3196588f049489ab5fjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:33:23,224] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee015df01c6b434b3196588f049489ab5fjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:33:23,283] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.483365ms
[2022-11-25 17:33:23,283] - [base_gsi:288] INFO - BUILD INDEX on default(employee015df01c6b434b3196588f049489ab5fjob_title,employee015df01c6b434b3196588f049489ab5fjoin_yr) USING GSI
[2022-11-25 17:33:24,316] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee015df01c6b434b3196588f049489ab5fjob_title,employee015df01c6b434b3196588f049489ab5fjoin_yr) USING GSI
[2022-11-25 17:33:24,348] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee015df01c6b434b3196588f049489ab5fjob_title%2Cemployee015df01c6b434b3196588f049489ab5fjoin_yr%29+USING+GSI
[2022-11-25 17:33:24,396] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.226325ms
[2022-11-25 17:33:25,431] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjob_title'
[2022-11-25 17:33:25,461] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjob_title%27
[2022-11-25 17:33:25,470] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.622789ms
[2022-11-25 17:33:26,502] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjob_title'
[2022-11-25 17:33:26,531] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjob_title%27
[2022-11-25 17:33:26,538] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.002729ms
[2022-11-25 17:33:26,566] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjoin_yr'
[2022-11-25 17:33:26,594] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjoin_yr%27
[2022-11-25 17:33:26,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.879789ms
[2022-11-25 17:33:27,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjob_title'
[2022-11-25 17:33:27,659] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjob_title%27
[2022-11-25 17:33:27,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.240115ms
[2022-11-25 17:33:27,698] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee015df01c6b434b3196588f049489ab5fjob_title ON default USING GSI
[2022-11-25 17:33:27,726] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee015df01c6b434b3196588f049489ab5fjob_title+ON+default+USING+GSI
[2022-11-25 17:33:27,773] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.737965ms
[2022-11-25 17:33:27,814] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjoin_yr'
[2022-11-25 17:33:27,844] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjoin_yr%27
[2022-11-25 17:33:27,852] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.625742ms
[2022-11-25 17:33:27,882] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee015df01c6b434b3196588f049489ab5fjoin_yr ON default USING GSI
[2022-11-25 17:33:27,910] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee015df01c6b434b3196588f049489ab5fjoin_yr+ON+default+USING+GSI
[2022-11-25 17:33:27,963] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.064031ms
[2022-11-25 17:33:27,997] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjob_title'
[2022-11-25 17:33:28,027] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjob_title%27
[2022-11-25 17:33:28,034] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.712441ms
[2022-11-25 17:33:28,035] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bfad4a8b-a892-403d-b31e-c93b900e9669', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.712441ms', 'executionTime': '5.627748ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:33:28,067] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015df01c6b434b3196588f049489ab5fjoin_yr'
[2022-11-25 17:33:28,097] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015df01c6b434b3196588f049489ab5fjoin_yr%27
[2022-11-25 17:33:28,101] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.731099ms
[2022-11-25 17:33:28,101] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '01a9fbf4-35d4-485f-a3d4-f8a0ad6362b9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.731099ms', 'executionTime': '1.64724ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:33:28,215] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:33:28,219] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:28,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:28,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:28,979] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:33:28,979] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:33:29,086] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:33:29,150] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:33:29,153] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:29,153] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:29,846] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:29,904] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:33:29,905] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:33:30,019] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:33:30,078] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:33:30,078] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:33:30,079] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:33:30,111] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:33:30,142] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:33:30,150] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.93875ms
[2022-11-25 17:33:30,179] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:33:30,208] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:33:30,266] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.451292ms
[2022-11-25 17:33:30,343] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:33:30,344] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.36498796992947, 'mem_free': 13119418368, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:33:30,344] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:33:30,349] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:30,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:31,042] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:31,047] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:31,047] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:31,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:31,973] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:31,973] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:33,155] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:33,166] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:33,166] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:34,356] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:41,649] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-11-25 17:33:41,799] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:33:43,063] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:33:43,094] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:33:43,094] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:33:43,159] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:33:43,216] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:33:43,270] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:33:43,271] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:33:43,354] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:33:43,356] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:33:43,384] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:43,534] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:33:43,534] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:43,563] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:33:43,591] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:33:43,591] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:43,620] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:33:43,648] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:33:43,648] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:43,676] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:33:43,703] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:33:43,703] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:43,732] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:33:43,732] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-11-25 17:33:43,732] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:33:43,733] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 89.299s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_8'}
[2022-11-25 17:33:43,831] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:43,831] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:44,517] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:44,554] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:33:44,649] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:33:44,650] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-11-25 17:33:44,650] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:33:44,860] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:33:44,890] - [task:165] INFO -  {'uptime': '692', 'memoryTotal': 15466930176, 'memoryFree': 13268545536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:33:44,921] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:33:44,921] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:33:44,922] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:33:44,955] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:33:44,997] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:33:44,997] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:33:45,029] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:33:45,030] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:33:45,030] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:33:45,030] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:33:45,086] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:33:45,090] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:45,090] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:45,789] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:45,790] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:33:45,926] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:33:45,927] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:33:45,961] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:45,991] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:46,025] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:33:46,166] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:33:46,200] - [task:165] INFO -  {'uptime': '688', 'memoryTotal': 15466930176, 'memoryFree': 13266198528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:33:46,232] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:33:46,266] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:33:46,266] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:33:46,327] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:33:46,331] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:46,331] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:47,026] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:47,027] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:33:47,166] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:33:47,167] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:33:47,200] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:47,230] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:47,263] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:33:47,392] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:33:47,423] - [task:165] INFO -  {'uptime': '689', 'memoryTotal': 15466930176, 'memoryFree': 13263663104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:33:47,453] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:33:47,485] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:33:47,485] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:33:47,542] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:33:47,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:47,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:48,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:48,247] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:33:48,381] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:33:48,382] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:33:48,416] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:48,446] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:48,477] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:33:48,609] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:33:48,639] - [task:165] INFO -  {'uptime': '688', 'memoryTotal': 15466930176, 'memoryFree': 13265420288, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:33:48,669] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:33:48,701] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:33:48,701] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:33:48,756] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:33:48,760] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:48,760] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:49,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:49,450] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:33:49,582] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:33:49,584] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:33:49,616] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:49,645] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:33:49,677] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:33:49,779] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:33:50,377] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:33:55,382] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:33:55,466] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:33:55,470] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:55,470] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:56,117] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:56,118] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:33:56,254] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:33:56,255] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:33:56,255] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:33:56,947] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:33:57,007] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:33:57,008] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:33:57,979] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:33:58,296] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:33:58,601] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:33:58,605] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-11-25 17:33:58,664] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:58,664] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:33:59,326] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:33:59,331] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:33:59,331] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:00,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:00,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:00,338] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:01,468] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:01,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:01,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:02,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:09,243] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:34:09,243] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.72508233421681, 'mem_free': 13285933056, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:34:09,244] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:34:09,244] - [basetestcase:466] INFO - Time to execute basesetup : 25.41609215736389
[2022-11-25 17:34:09,299] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:34:09,299] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:34:09,355] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:34:09,355] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:34:09,412] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:34:09,412] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:34:09,468] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:34:09,468] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:34:09,520] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:09,585] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:34:09,585] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:34:09,586] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:34:14,596] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:34:14,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:14,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:15,225] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:16,317] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:34:16,648] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:34:19,728] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:34:19,811] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:19,811] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:34:19,812] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:34:49,832] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:34:49,863] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:34:49,892] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:34:49,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.138156ms
[2022-11-25 17:34:49,968] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c4909761-864d-43d8-a7aa-e4132b2ac7d1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '73.138156ms', 'executionTime': '73.058138ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:34:49,968] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:34:49,995] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:34:50,020] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:34:50,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 714.582648ms
[2022-11-25 17:34:50,737] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:34:50,804] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:34:50,844] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:34:50,853] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.975382ms
[2022-11-25 17:34:51,051] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:51,091] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:34:51,107] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:34:51,108] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:34:51,120] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:34:51,121] - [base_gsi:332] INFO - []
[2022-11-25 17:34:52,018] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:34:52,047] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:34:52,122] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.124534ms
[2022-11-25 17:34:52,123] - [base_gsi:288] INFO - BUILD INDEX on default(employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr) USING GSI
[2022-11-25 17:34:53,154] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr) USING GSI
[2022-11-25 17:34:53,181] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%29+USING+GSI
[2022-11-25 17:34:53,209] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.307316ms
[2022-11-25 17:34:54,245] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr'
[2022-11-25 17:34:54,272] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%27
[2022-11-25 17:34:54,283] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.837013ms
[2022-11-25 17:34:55,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr'
[2022-11-25 17:34:55,341] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%27
[2022-11-25 17:34:55,348] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.315029ms
[2022-11-25 17:34:56,379] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr'
[2022-11-25 17:34:56,405] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%27
[2022-11-25 17:34:56,416] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.102043ms
[2022-11-25 17:34:56,443] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr ON default USING GSI
[2022-11-25 17:34:56,468] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr+ON+default+USING+GSI
[2022-11-25 17:34:56,506] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.742759ms
[2022-11-25 17:34:56,547] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr'
[2022-11-25 17:34:56,575] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7608ce1156e34526bd76d5b28b8157d3job_title_join_yr%27
[2022-11-25 17:34:56,582] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.887324ms
[2022-11-25 17:34:56,583] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ec769b78-bc3b-49ef-948f-7fd69c011aba', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.887324ms', 'executionTime': '5.809891ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:34:56,689] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:56,692] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:56,693] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:57,346] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:57,401] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:34:57,401] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:34:57,509] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:34:57,568] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:57,571] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:57,571] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:58,272] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:58,332] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:34:58,332] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:34:58,449] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:34:58,510] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:34:58,510] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:34:58,510] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:34:58,539] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:34:58,570] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:34:58,581] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.846568ms
[2022-11-25 17:34:58,610] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:34:58,639] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:34:58,693] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.382292ms
[2022-11-25 17:34:58,762] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:34:58,762] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.886192091801123, 'mem_free': 13154000896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:34:58,762] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:34:58,767] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:58,767] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:34:59,505] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:34:59,510] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:34:59,510] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:00,224] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:00,228] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:00,229] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:01,355] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:01,364] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:01,364] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:02,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:09,862] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-11-25 17:35:10,008] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:35:10,596] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:35:10,625] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:35:10,625] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:35:10,682] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:35:10,736] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:35:10,789] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:35:10,790] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:35:10,872] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:35:10,873] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:35:10,898] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:11,028] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:35:11,028] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:11,054] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:35:11,081] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:35:11,081] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:11,108] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:35:11,143] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:35:11,143] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:11,170] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:35:11,196] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:35:11,196] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:11,223] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:35:11,223] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-11-25 17:35:11,223] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:35:11,224] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 87.450s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_9'}
[2022-11-25 17:35:11,317] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:11,317] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:11,989] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:12,024] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:35:12,104] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:35:12,104] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-11-25 17:35:12,105] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:35:12,347] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:35:12,375] - [task:165] INFO -  {'uptime': '779', 'memoryTotal': 15466930176, 'memoryFree': 13278261248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:35:12,402] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:35:12,402] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:35:12,402] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:35:12,432] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:35:12,468] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:35:12,468] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:35:12,496] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:35:12,497] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:35:12,497] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:35:12,498] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:35:12,545] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:35:12,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:12,549] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:13,171] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:13,172] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:35:13,299] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:35:13,300] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:35:13,331] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:13,357] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:13,386] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:35:13,512] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:35:13,539] - [task:165] INFO -  {'uptime': '778', 'memoryTotal': 15466930176, 'memoryFree': 13277917184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:35:13,565] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:35:13,592] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:35:13,593] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:35:13,644] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:35:13,647] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:13,647] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:14,323] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:14,325] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:35:14,455] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:35:14,456] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:35:14,487] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:14,513] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:14,543] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:35:14,667] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:35:14,695] - [task:165] INFO -  {'uptime': '779', 'memoryTotal': 15466930176, 'memoryFree': 13276561408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:35:14,722] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:35:14,751] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:35:14,751] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:35:14,803] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:35:14,808] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:14,809] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:15,491] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:15,492] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:35:15,639] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:35:15,640] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:35:15,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:15,698] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:15,727] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:35:15,846] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:35:15,873] - [task:165] INFO -  {'uptime': '778', 'memoryTotal': 15466930176, 'memoryFree': 13276024832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:35:15,899] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:35:15,927] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:35:15,927] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:35:15,979] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:35:15,983] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:15,983] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:16,624] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:16,625] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:35:16,754] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:35:16,755] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:35:16,785] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:16,810] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:35:16,839] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:35:16,934] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:35:17,458] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:35:22,462] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:35:22,549] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:35:22,553] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:22,553] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:23,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:23,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:35:23,383] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:35:23,384] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:35:23,385] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:35:24,052] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:35:24,109] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:35:24,110] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:35:25,028] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:35:25,354] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:35:25,667] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:35:25,672] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-11-25 17:35:25,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:25,744] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:26,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:26,387] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:26,387] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:27,399] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:27,407] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:27,408] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:28,511] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:28,520] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:28,520] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:29,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:36,613] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:35:36,614] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.50525264657482, 'mem_free': 13276921856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:35:36,614] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:35:36,614] - [basetestcase:466] INFO - Time to execute basesetup : 25.29998207092285
[2022-11-25 17:35:36,668] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:35:36,668] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:35:36,729] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:35:36,729] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:35:36,789] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:35:36,789] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:35:36,844] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:35:36,845] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:35:36,896] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:35:36,968] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:35:36,969] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:35:36,969] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:35:41,981] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:35:41,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:35:41,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:35:42,607] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:35:43,710] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:35:44,043] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:35:47,120] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:35:47,205] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:35:47,206] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:35:47,206] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:36:17,221] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:36:17,251] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:36:17,277] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:36:17,347] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.299553ms
[2022-11-25 17:36:17,347] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '92c48e20-4dc5-4f6d-a26d-e17a9f949c41', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.299553ms', 'executionTime': '68.218617ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:36:17,348] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:36:17,374] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:36:17,399] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:36:18,076] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 674.537834ms
[2022-11-25 17:36:18,076] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:36:18,147] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:36:18,181] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:36:18,191] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.253367ms
[2022-11-25 17:36:18,384] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:18,423] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:36:18,464] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:36:18,465] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:36:18,486] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:36:18,548] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:19,350] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8c94278224454365bd285470e1ecb5a8join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:36:19,376] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee8c94278224454365bd285470e1ecb5a8join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:36:19,452] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.215443ms
[2022-11-25 17:36:19,453] - [base_gsi:288] INFO - BUILD INDEX on default(`employee8c94278224454365bd285470e1ecb5a8join_yr`) USING GSI
[2022-11-25 17:36:20,487] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee8c94278224454365bd285470e1ecb5a8join_yr`) USING GSI
[2022-11-25 17:36:20,516] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee8c94278224454365bd285470e1ecb5a8join_yr%60%29+USING+GSI
[2022-11-25 17:36:20,541] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.265383ms
[2022-11-25 17:36:20,588] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8c94278224454365bd285470e1ecb5a8join_yr'
[2022-11-25 17:36:20,620] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8c94278224454365bd285470e1ecb5a8join_yr%27
[2022-11-25 17:36:20,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.934587ms
[2022-11-25 17:36:21,663] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8c94278224454365bd285470e1ecb5a8join_yr'
[2022-11-25 17:36:21,691] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8c94278224454365bd285470e1ecb5a8join_yr%27
[2022-11-25 17:36:21,702] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.217575ms
[2022-11-25 17:36:22,734] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8c94278224454365bd285470e1ecb5a8join_yr'
[2022-11-25 17:36:22,762] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8c94278224454365bd285470e1ecb5a8join_yr%27
[2022-11-25 17:36:22,772] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.987578ms
[2022-11-25 17:36:23,804] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:36:23,831] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-11-25 17:36:23,836] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.051731ms
[2022-11-25 17:36:23,836] - [base_gsi:510] INFO - {'requestID': 'bc67ded7-b9ee-4633-ba39-81f592a19511', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8c94278224454365bd285470e1ecb5a8join_yr', 'index_id': 'c9460cfb19585c1f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.051731ms', 'executionTime': '2.968649ms', 'resultCount': 1, 'resultSize': 948, 'serviceLoad': 6}}
[2022-11-25 17:36:23,837] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:36:23,837] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:36:23,837] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:36:23,838] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:36:23,839] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:36:23,839] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:36:23,840] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:36:23,840] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:36:23,897] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:36:23,937] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:36:23,966] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-11-25 17:36:24,217] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 237.428685ms
[2022-11-25 17:36:24,218] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:36:24,218] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:36:26,749] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:26,780] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8c94278224454365bd285470e1ecb5a8join_yr'
[2022-11-25 17:36:26,807] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8c94278224454365bd285470e1ecb5a8join_yr%27
[2022-11-25 17:36:26,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.194281ms
[2022-11-25 17:36:26,814] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8f9216a6-93e1-4159-88a9-a04e1cc0ce33', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.194281ms', 'executionTime': '5.117927ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:36:26,918] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:26,921] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:26,921] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:27,725] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:27,785] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:36:27,785] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:36:27,898] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:36:27,955] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:27,958] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:27,958] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:28,667] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:28,723] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:36:28,723] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:36:28,835] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:36:28,893] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:36:28,894] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:36:28,894] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:36:28,919] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:36:28,947] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:36:28,953] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.996302ms
[2022-11-25 17:36:28,953] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f60098d6-0a1f-4744-a990-ef6ebc9c10ba', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.996302ms', 'executionTime': '3.918036ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:36:29,006] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:36:29,006] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.109237699049893, 'mem_free': 13136117760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:36:29,007] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:36:29,010] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:29,010] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:29,720] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:29,725] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:29,725] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:30,453] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:30,458] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:30,458] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:31,515] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:31,526] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:31,527] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:32,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:40,915] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-11-25 17:36:41,060] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:36:41,118] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:36:41,174] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:36:41,231] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:36:41,231] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:36:41,308] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:36:41,309] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:36:41,335] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:41,472] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:36:41,472] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:41,502] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:36:41,531] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:36:41,531] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:41,563] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:36:41,593] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:36:41,593] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:41,623] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:36:41,650] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:36:41,650] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:41,678] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:36:41,678] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-11-25 17:36:41,678] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:36:41,679] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 90.423s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_10'}
[2022-11-25 17:36:41,778] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:41,778] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:42,501] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:42,536] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:36:42,625] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:36:42,625] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-11-25 17:36:42,626] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:36:42,805] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:36:42,832] - [task:165] INFO -  {'uptime': '870', 'memoryTotal': 15466930176, 'memoryFree': 13274509312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:36:42,858] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:36:42,858] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:36:42,859] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:36:42,895] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:36:42,929] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:36:42,930] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:36:42,958] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:36:42,959] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:36:42,959] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:36:42,960] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:36:43,010] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:36:43,014] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:43,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:43,698] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:43,699] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:36:43,829] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:36:43,830] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:36:43,860] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:43,886] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:43,917] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:36:44,050] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:36:44,079] - [task:165] INFO -  {'uptime': '869', 'memoryTotal': 15466930176, 'memoryFree': 13274812416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:36:44,108] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:36:44,139] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:36:44,139] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:36:44,192] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:36:44,197] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:44,197] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:44,890] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:44,891] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:36:45,021] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:36:45,022] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:36:45,053] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:45,079] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:45,107] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:36:45,225] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:36:45,254] - [task:165] INFO -  {'uptime': '869', 'memoryTotal': 15466930176, 'memoryFree': 13272842240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:36:45,281] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:36:45,311] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:36:45,311] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:36:45,364] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:36:45,367] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:45,367] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:46,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:46,095] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:36:46,231] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:36:46,233] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:36:46,262] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:46,288] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:46,317] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:36:46,436] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:36:46,464] - [task:165] INFO -  {'uptime': '869', 'memoryTotal': 15466930176, 'memoryFree': 13267533824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:36:46,490] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:36:46,519] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:36:46,519] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:36:46,572] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:36:46,576] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:46,576] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:47,305] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:47,306] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:36:47,442] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:36:47,443] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:36:47,475] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:47,504] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:36:47,536] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:36:47,634] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:36:48,176] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:36:53,182] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:36:53,271] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:36:53,275] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:53,275] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:53,994] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:53,995] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:36:54,129] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:36:54,130] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:36:54,131] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:36:54,786] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:36:54,846] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:36:54,847] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:36:55,796] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:36:56,137] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:36:56,465] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:36:56,469] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-11-25 17:36:56,534] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:56,534] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:57,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:57,246] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:57,246] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:58,286] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:58,295] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:58,295] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:36:59,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:36:59,497] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:36:59,498] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:37:00,730] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:37:08,198] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:37:08,199] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.78593777984534, 'mem_free': 13268074496, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:37:08,199] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:37:08,199] - [basetestcase:466] INFO - Time to execute basesetup : 26.42414879798889
[2022-11-25 17:37:08,252] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:37:08,252] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:37:08,312] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:37:08,312] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:37:08,374] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:37:08,374] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:37:08,429] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:37:08,429] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:37:08,483] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:37:08,547] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:37:08,548] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:37:08,548] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:37:13,558] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:37:13,562] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:37:13,562] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:37:14,273] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:37:15,381] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:37:15,554] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:37:18,644] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:37:18,726] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:37:18,726] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:37:18,727] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:37:48,755] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:37:48,787] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:37:48,815] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:37:48,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 75.940265ms
[2022-11-25 17:37:48,894] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2cf0ccde-9eff-44f0-bdf7-9e2227a8aeb1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '75.940265ms', 'executionTime': '75.847144ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:37:48,894] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:37:48,922] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:37:48,951] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:37:49,704] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 750.792096ms
[2022-11-25 17:37:49,704] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:37:49,810] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:37:49,850] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:37:49,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.712303ms
[2022-11-25 17:37:50,072] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:37:50,109] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:37:50,126] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:37:50,132] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:37:50,147] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:37:50,236] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:37:51,030] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef4e5f6db811743139da90def5b59a1d7join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:37:51,057] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employeef4e5f6db811743139da90def5b59a1d7join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:37:51,304] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 244.328664ms
[2022-11-25 17:37:51,304] - [base_gsi:288] INFO - BUILD INDEX on default(`employeef4e5f6db811743139da90def5b59a1d7join_yr`) USING GSI
[2022-11-25 17:37:52,342] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeef4e5f6db811743139da90def5b59a1d7join_yr`) USING GSI
[2022-11-25 17:37:52,371] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeef4e5f6db811743139da90def5b59a1d7join_yr%60%29+USING+GSI
[2022-11-25 17:37:52,424] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.416319ms
[2022-11-25 17:37:52,467] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef4e5f6db811743139da90def5b59a1d7join_yr'
[2022-11-25 17:37:52,513] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef4e5f6db811743139da90def5b59a1d7join_yr%27
[2022-11-25 17:37:52,523] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.621179ms
[2022-11-25 17:37:53,557] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef4e5f6db811743139da90def5b59a1d7join_yr'
[2022-11-25 17:37:53,586] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef4e5f6db811743139da90def5b59a1d7join_yr%27
[2022-11-25 17:37:53,596] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.249356ms
[2022-11-25 17:37:54,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef4e5f6db811743139da90def5b59a1d7join_yr'
[2022-11-25 17:37:54,659] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef4e5f6db811743139da90def5b59a1d7join_yr%27
[2022-11-25 17:37:54,670] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.790036ms
[2022-11-25 17:37:55,702] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:37:55,729] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-11-25 17:37:55,734] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.870349ms
[2022-11-25 17:37:55,734] - [base_gsi:510] INFO - {'requestID': 'ceffb1fa-dc6f-4cf7-aa46-939e1e727e7e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef4e5f6db811743139da90def5b59a1d7join_yr', 'index_id': '4e6b9b23a0917e66', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.870349ms', 'executionTime': '3.800316ms', 'resultCount': 1, 'resultSize': 948, 'serviceLoad': 6}}
[2022-11-25 17:37:55,735] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:37:55,735] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:55,736] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:37:55,736] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:37:55,736] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:55,737] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:55,737] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:55,737] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:55,796] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:55,836] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:37:55,863] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-11-25 17:37:56,038] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 161.709176ms
[2022-11-25 17:37:56,038] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:37:56,039] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:37:57,261] - [on_prem_rest_client:3095] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-11-25 17:37:57,318] - [on_prem_rest_client:3103] INFO - bucket default updated
[2022-11-25 17:37:57,355] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:37:57,384] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-11-25 17:37:57,388] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.074186ms
[2022-11-25 17:37:57,388] - [base_gsi:510] INFO - {'requestID': 'f899291d-f418-4102-a409-100b1e78e914', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef4e5f6db811743139da90def5b59a1d7join_yr', 'index_id': '4e6b9b23a0917e66', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.074186ms', 'executionTime': '2.006716ms', 'resultCount': 1, 'resultSize': 948, 'serviceLoad': 6}}
[2022-11-25 17:37:57,389] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:37:57,389] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:57,389] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:37:57,389] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:37:57,390] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:57,390] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:57,390] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:57,391] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:37:57,446] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:37:57,486] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:37:57,512] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-11-25 17:37:57,668] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 145.672881ms
[2022-11-25 17:37:57,668] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:37:57,669] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:37:58,844] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef4e5f6db811743139da90def5b59a1d7join_yr ON default USING GSI
[2022-11-25 17:37:58,871] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+INDEX+employeef4e5f6db811743139da90def5b59a1d7join_yr+ON+default+USING+GSI
[2022-11-25 17:37:58,938] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.953665ms
[2022-11-25 17:37:58,971] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef4e5f6db811743139da90def5b59a1d7join_yr'
[2022-11-25 17:37:58,997] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef4e5f6db811743139da90def5b59a1d7join_yr%27
[2022-11-25 17:37:59,006] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.616668ms
[2022-11-25 17:37:59,006] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4a66cd70-bbe4-425d-bd58-b8c123b11d3b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.616668ms', 'executionTime': '6.526461ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:37:59,116] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:37:59,120] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:37:59,120] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:37:59,872] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:37:59,931] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:37:59,931] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:38:00,048] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:38:00,112] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:38:00,115] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:00,116] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:00,881] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:00,938] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:38:00,939] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:38:01,059] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:38:01,120] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:38:01,120] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:38:01,120] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:38:01,148] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:38:01,177] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:38:01,188] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.400832ms
[2022-11-25 17:38:01,216] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:38:01,244] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:38:01,309] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.982811ms
[2022-11-25 17:38:01,376] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:38:01,376] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.100055838291349, 'mem_free': 13113282560, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:38:01,376] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:38:01,380] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:01,380] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:02,139] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:02,144] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:02,144] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:03,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:03,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:03,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:04,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:04,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:04,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:05,876] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:13,392] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-11-25 17:38:13,548] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:38:14,349] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:38:14,379] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:38:14,379] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:38:14,436] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:38:14,492] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:38:14,547] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:38:14,548] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:38:14,634] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:38:14,635] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:38:14,673] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:14,816] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:38:14,817] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:14,845] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:38:14,875] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:38:14,875] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:14,904] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:38:14,931] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:38:14,931] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:14,959] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:38:14,985] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:38:14,986] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:15,013] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:38:15,013] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-11-25 17:38:15,014] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:38:15,015] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 93.294s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_11'}
[2022-11-25 17:38:15,102] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:15,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:15,852] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:15,888] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:38:15,972] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:38:15,972] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-11-25 17:38:15,973] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:38:16,130] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:38:16,158] - [task:165] INFO -  {'uptime': '963', 'memoryTotal': 15466930176, 'memoryFree': 13234876416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:38:16,185] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:38:16,185] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:38:16,186] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:38:16,217] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:38:16,254] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:38:16,255] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:38:16,284] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:38:16,285] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:38:16,285] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:38:16,286] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:38:16,335] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:38:16,338] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:16,338] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:17,090] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:17,092] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:38:17,233] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:38:17,234] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:38:17,265] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:17,293] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:17,323] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:38:17,458] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:38:17,486] - [task:165] INFO -  {'uptime': '964', 'memoryTotal': 15466930176, 'memoryFree': 13232947200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:38:17,512] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:38:17,542] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:38:17,542] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:38:17,597] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:38:17,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:17,600] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:18,353] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:18,354] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:38:18,492] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:38:18,493] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:38:18,525] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:18,551] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:18,580] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:38:18,710] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:38:18,738] - [task:165] INFO -  {'uptime': '960', 'memoryTotal': 15466930176, 'memoryFree': 13233328128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:38:18,765] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:38:18,796] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:38:18,796] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:38:18,849] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:38:18,854] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:18,855] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:19,596] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:19,597] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:38:19,732] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:38:19,733] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:38:19,763] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:19,790] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:19,819] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:38:19,939] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:38:19,967] - [task:165] INFO -  {'uptime': '959', 'memoryTotal': 15466930176, 'memoryFree': 13233864704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:38:19,993] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:38:20,022] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:38:20,023] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:38:20,079] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:38:20,082] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:20,082] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:20,840] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:20,841] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:38:20,980] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:38:20,981] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:38:21,013] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:21,040] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:38:21,070] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:38:21,162] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:38:21,723] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:38:26,729] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:38:26,820] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:38:26,825] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:26,825] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:27,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:27,596] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:38:27,733] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:38:27,735] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:38:27,735] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:38:28,365] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:38:28,428] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:38:28,428] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:38:29,382] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:38:29,705] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:38:29,994] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:38:29,997] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-11-25 17:38:30,056] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:30,057] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:30,827] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:30,832] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:30,832] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:32,005] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:32,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:32,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:33,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:33,310] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:33,310] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:34,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:42,152] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:38:42,153] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.90850103310125, 'mem_free': 13230538752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:38:42,153] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:38:42,153] - [basetestcase:466] INFO - Time to execute basesetup : 27.054182052612305
[2022-11-25 17:38:42,209] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:38:42,209] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:38:42,270] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:38:42,270] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:38:42,329] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:38:42,329] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:38:42,388] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:38:42,388] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:38:42,445] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:38:42,509] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:38:42,510] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:38:42,510] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:38:47,523] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:38:47,526] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:38:47,526] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:38:48,260] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:38:49,382] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:38:49,737] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:38:52,663] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:38:52,749] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:38:52,749] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:38:52,749] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:39:22,780] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:39:22,810] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:39:22,836] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:39:22,909] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.767385ms
[2022-11-25 17:39:22,909] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2b6bd996-ddc8-4c8a-b2f8-01b6fe9a0c46', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.767385ms', 'executionTime': '70.694384ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:39:22,910] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:39:22,938] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:39:22,965] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:39:23,711] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 743.418215ms
[2022-11-25 17:39:23,711] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:39:23,789] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:39:23,830] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:39:23,838] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.848642ms
[2022-11-25 17:39:24,048] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:24,086] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:39:24,107] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:39:24,109] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:39:24,128] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:39:24,185] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:25,007] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee00e42590c2d8483888feed0f4f25a39ajoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-11-25 17:39:25,034] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+INDEX+%60employee00e42590c2d8483888feed0f4f25a39ajoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-11-25 17:39:25,119] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 82.660936ms
[2022-11-25 17:39:25,119] - [base_gsi:288] INFO - BUILD INDEX on default(`employee00e42590c2d8483888feed0f4f25a39ajoin_yr`) USING GSI
[2022-11-25 17:39:26,151] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee00e42590c2d8483888feed0f4f25a39ajoin_yr`) USING GSI
[2022-11-25 17:39:26,178] - [on_prem_rest_client:4212] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee00e42590c2d8483888feed0f4f25a39ajoin_yr%60%29+USING+GSI
[2022-11-25 17:39:26,207] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.494735ms
[2022-11-25 17:39:26,252] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr'
[2022-11-25 17:39:26,291] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00e42590c2d8483888feed0f4f25a39ajoin_yr%27
[2022-11-25 17:39:26,300] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.223697ms
[2022-11-25 17:39:27,332] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr'
[2022-11-25 17:39:27,361] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00e42590c2d8483888feed0f4f25a39ajoin_yr%27
[2022-11-25 17:39:27,370] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.829839ms
[2022-11-25 17:39:28,402] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr'
[2022-11-25 17:39:28,428] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00e42590c2d8483888feed0f4f25a39ajoin_yr%27
[2022-11-25 17:39:28,438] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.696608ms
[2022-11-25 17:39:29,469] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:39:29,498] - [on_prem_rest_client:4212] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-11-25 17:39:29,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.356467ms
[2022-11-25 17:39:29,503] - [base_gsi:510] INFO - {'requestID': '3fca899a-ef06-42e3-893a-0fdbb50207aa', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr', 'index_id': 'f6d4bcbc7b7fe403', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'preserve_order': True, 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.356467ms', 'executionTime': '3.270448ms', 'resultCount': 1, 'resultSize': 948, 'serviceLoad': 6}}
[2022-11-25 17:39:29,504] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-11-25 17:39:29,504] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:39:29,504] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-11-25 17:39:29,504] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-11-25 17:39:29,505] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:39:29,505] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:39:29,505] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:39:29,505] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-11-25 17:39:29,562] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-11-25 17:39:29,601] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-11-25 17:39:29,627] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-11-25 17:39:29,841] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 199.608733ms
[2022-11-25 17:39:29,842] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-11-25 17:39:29,842] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-11-25 17:39:32,051] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-11-25 17:39:34,365] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:39:34,426] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:39:34,427] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:39:35,351] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:39:35,662] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:39:35,924] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:39:35,927] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-11-25 17:39:37,962] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:39:37,962] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:37,992] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:39:38,021] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:39:38,022] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:38,050] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:39:38,080] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:39:38,080] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:38,107] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:39:38,137] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:39:38,137] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:38,166] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:39:38,223] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:38,248] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr'
[2022-11-25 17:39:38,274] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00e42590c2d8483888feed0f4f25a39ajoin_yr%27
[2022-11-25 17:39:38,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.548247ms
[2022-11-25 17:39:38,323] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1f1b64f2-e712-4344-b3d9-1d7943e259d4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '46.548247ms', 'executionTime': '46.47059ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:39:38,351] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee00e42590c2d8483888feed0f4f25a39ajoin_yr'
[2022-11-25 17:39:38,377] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee00e42590c2d8483888feed0f4f25a39ajoin_yr%27
[2022-11-25 17:39:38,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.342362ms
[2022-11-25 17:39:38,380] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a6936d1a-2403-48b6-a885-8d341d3725bf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.342362ms', 'executionTime': '1.266133ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:39:38,432] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:38,557] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:38,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:38,560] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:39,276] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:39,335] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:39:39,335] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:39:39,455] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:39:39,511] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:39,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:39,514] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:40,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:40,322] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:39:40,323] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:39:40,445] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:39:40,526] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:39:40,527] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:39:40,527] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:39:40,568] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:39:40,597] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:39:40,605] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.319974ms
[2022-11-25 17:39:40,606] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c54dc5ac-d64f-4107-904b-2c4beb513a94', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.319974ms', 'executionTime': '7.241535ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:39:40,606] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:39:40,632] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:39:40,658] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:39:40,661] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.229209ms
[2022-11-25 17:39:40,661] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '325c4850-c171-4314-904e-f571689e7fd1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.229209ms', 'executionTime': '1.154097ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:39:40,712] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:39:40,713] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.762594703589, 'mem_free': 13212594176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:39:40,713] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:39:40,716] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:40,717] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:41,443] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:41,448] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:41,448] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:42,345] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:42,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:42,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:43,653] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:43,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:43,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:44,973] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:53,165] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-11-25 17:39:53,320] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:39:53,641] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:39:53,671] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:39:53,672] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:39:53,733] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:39:53,793] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:39:53,851] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:39:53,852] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:39:53,942] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:39:53,943] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:39:53,976] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:54,111] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:39:54,111] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:54,139] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:39:54,166] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:39:54,166] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:54,193] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:39:54,220] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:39:54,220] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:54,248] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:39:54,275] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:39:54,275] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:54,302] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:39:54,302] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-11-25 17:39:54,302] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:39:54,303] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 99.258s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Nov-25_17-22-08/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-11-25 17:39:54,363] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:54,364] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:55,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:55,181] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:39:55,264] - [on_prem_rest_client:2679] INFO - Node version in cluster 7.5.0-3184-rel-EE-enterprise
[2022-11-25 17:39:55,264] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-11-25 17:39:55,265] - [basetestcase:223] INFO - initializing cluster
[2022-11-25 17:39:55,391] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-11-25 17:39:55,418] - [task:165] INFO -  {'uptime': '1062', 'memoryTotal': 15466930176, 'memoryFree': 13213581312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:39:55,444] - [task:204] INFO - quota for index service will be 256 MB
[2022-11-25 17:39:55,445] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-11-25 17:39:55,445] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-11-25 17:39:55,484] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-11-25 17:39:55,518] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-11-25 17:39:55,519] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-11-25 17:39:55,549] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-11-25 17:39:55,550] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-11-25 17:39:55,550] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-11-25 17:39:55,551] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-11-25 17:39:55,608] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:39:55,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:55,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:56,383] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:56,384] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:39:56,525] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:39:56,526] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:39:56,561] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:56,590] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:56,619] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:39:56,744] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-11-25 17:39:56,772] - [task:165] INFO -  {'uptime': '1059', 'memoryTotal': 15466930176, 'memoryFree': 13209317376, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:39:56,799] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:39:56,829] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-11-25 17:39:56,830] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-11-25 17:39:56,884] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:39:56,887] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:56,887] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:57,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:57,659] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:39:57,812] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:39:57,813] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:39:57,845] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:57,872] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:57,909] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:39:58,034] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-11-25 17:39:58,063] - [task:165] INFO -  {'uptime': '1060', 'memoryTotal': 15466930176, 'memoryFree': 13213163520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:39:58,090] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:39:58,120] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-11-25 17:39:58,120] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-11-25 17:39:58,174] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:39:58,177] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:58,177] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:39:58,963] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:39:58,964] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:39:59,121] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:39:59,122] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:39:59,154] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:59,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:39:59,213] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:39:59,340] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-11-25 17:39:59,370] - [task:165] INFO -  {'uptime': '1060', 'memoryTotal': 15466930176, 'memoryFree': 13212696576, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.5.0-3184-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-11-25 17:39:59,398] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-11-25 17:39:59,429] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-11-25 17:39:59,429] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-11-25 17:39:59,485] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-11-25 17:39:59,488] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:39:59,488] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:00,277] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:00,278] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:40:00,423] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:40:00,424] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:40:00,458] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:40:00,488] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-11-25 17:40:00,521] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-11-25 17:40:00,625] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-11-25 17:40:01,367] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:40:06,371] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-11-25 17:40:06,459] - [basetestcase:261] INFO - done initializing cluster
[2022-11-25 17:40:06,464] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:06,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:07,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:07,271] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-11-25 17:40:07,414] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-11-25 17:40:07,415] - [remote_util:5249] INFO - b'ok'
[2022-11-25 17:40:07,416] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-11-25 17:40:07,792] - [on_prem_rest_client:3033] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-11-25 17:40:07,847] - [on_prem_rest_client:3058] INFO - 0.06 seconds to create bucket default
[2022-11-25 17:40:07,848] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-11-25 17:40:08,775] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:40:09,091] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:40:09,391] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-11-25 17:40:09,395] - [basetestcase:434] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-11-25 17:40:09,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:09,467] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:10,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:10,242] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:10,242] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:11,408] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:11,416] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:11,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:12,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:12,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:12,649] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:13,892] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:21,663] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:40:21,664] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.3486797562627, 'mem_free': 13221801984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:40:21,664] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:40:21,664] - [basetestcase:466] INFO - Time to execute basesetup : 27.303640604019165
[2022-11-25 17:40:21,722] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:40:21,722] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:40:21,781] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:40:21,781] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:40:21,839] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:40:21,840] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:40:21,898] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-11-25 17:40:21,898] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-11-25 17:40:21,952] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:40:22,023] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-11-25 17:40:22,023] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-11-25 17:40:22,023] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-11-25 17:40:27,036] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-11-25 17:40:27,039] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:40:27,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:40:27,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:40:29,173] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-11-25 17:40:29,358] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-11-25 17:40:32,139] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-11-25 17:40:32,222] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:40:32,222] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-11-25 17:40:32,222] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-11-25 17:41:02,247] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-11-25 17:41:02,280] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:41:02,308] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:41:02,383] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 73.123275ms
[2022-11-25 17:41:02,383] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4517ada1-4ddc-41ca-ae69-f28a41b26ab5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '73.123275ms', 'executionTime': '73.035973ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-11-25 17:41:02,384] - [tuq_helper:758] INFO - Create primary index
[2022-11-25 17:41:02,412] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-11-25 17:41:02,440] - [on_prem_rest_client:4212] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-11-25 17:41:03,703] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.261326337s
[2022-11-25 17:41:03,704] - [tuq_helper:760] INFO - Check if index is online
[2022-11-25 17:41:03,755] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:41:03,793] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:41:03,803] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.533965ms
[2022-11-25 17:41:04,023] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:41:04,060] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-11-25 17:41:04,079] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-11-25 17:41:04,080] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-11-25 17:41:04,094] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-11-25 17:41:04,166] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:41:04,305] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:41:04,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:04,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:05,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:05,238] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:41:05,238] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-11-25 17:41:05,375] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:41:05,433] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:41:05,437] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:05,437] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:06,282] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:06,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-11-25 17:41:06,343] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-11-25 17:41:06,476] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-11-25 17:41:06,535] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-11-25 17:41:06,535] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-11-25 17:41:06,535] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:41:06,562] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-11-25 17:41:06,589] - [on_prem_rest_client:4212] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-11-25 17:41:06,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.900561ms
[2022-11-25 17:41:06,628] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-11-25 17:41:06,656] - [on_prem_rest_client:4212] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-11-25 17:41:06,931] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 273.619598ms
[2022-11-25 17:41:06,995] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-11-25 17:41:06,996] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.939673467208065, 'mem_free': 13043630080, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-11-25 17:41:06,996] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-11-25 17:41:07,000] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:07,000] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:07,845] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:07,851] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:07,851] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:08,707] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:08,712] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:08,713] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:10,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:10,090] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-11-25 17:41:10,091] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-11-25 17:41:11,530] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-11-25 17:41:20,974] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-11-25 17:41:21,135] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-11-25 17:41:21,600] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-11-25 17:41:21,631] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-11-25 17:41:21,631] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-11-25 17:41:21,695] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:41:21,753] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:41:21,815] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-11-25 17:41:21,816] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-11-25 17:41:21,907] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-11-25 17:41:21,909] - [basetestcase:742] INFO - b'"User was not found."'
[2022-11-25 17:41:21,937] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:41:22,091] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-11-25 17:41:22,092] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:41:22,123] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-11-25 17:41:22,152] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-11-25 17:41:22,152] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:41:22,184] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-11-25 17:41:22,214] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-11-25 17:41:22,214] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:41:22,249] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-11-25 17:41:22,278] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-11-25 17:41:22,279] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-11-25 17:41:22,309] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-11-25 17:41:22,309] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-11-25 17:41:22,310] - [basetestcase:773] INFO - closing all ssh connections
[2022-11-25 17:41:22,310] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 88.053s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 17150 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-25.11.2022-14.42.cfg
Build Log: make-25.11.2022-14.42.log
Server Log: logs-25.11.2022-14.42.tar.gz

Finished