Building

Started building at 2022/06/12 09:30:56
Using pegged server, 1350 build
Calculating base
Updating mirror
Basing run on 7.2.0-1350 932b6b5a1
Updating tree for run 12.06.2022-09.30
query is at b2c39af, changes since last good build: none
gometa is at 8625ed1, changes since last good build: none
ns_server is at 717003b, changes since last good build: none
couchstore is at f0c688c, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 79f882c, changes since last good build: none
Switching indexing to unstable
indexing is at d10013e, changes since last good build: none
Switching plasma to unstable
plasma is at b6bd8f8, changes since last good build: 
 b6bd8f8 test: Add short run for GSI test
Switching nitro to unstable
nitro is at cb16486, changes since last good build: 
fatal: Invalid revision range 1276e501d0b479d13ef033e6d16c021ec81b19ac..HEAD

Switching gometa to master
gometa is at 8625ed1, changes since last good build: none
Switching testrunner to master
testrunner is at 8cb9a85, changes since last good build: none
Pulling in uncommitted change 174974 at refs/changes/74/174974/2
Total 14 (delta 12), reused 14 (delta 12)
[unstable 573fc318] MB-51825: VariableNumVBuckets-1: Add numVBuckets to ClusterInfoProvider
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed May 18 10:19:47 2022 +0530
 4 files changed, 60 insertions(+)
Pulling in uncommitted change 175575 at refs/changes/75/175575/2
Total 5 (delta 4), reused 5 (delta 4)
[unstable c539ac57] Revert "MB-51425 Keep minNumShard consistent with max_cpu_percent unless changed explicitly"
 Author: varun.velamuri 
 Date: Thu Jun 9 10:56:33 2022 +0000
 1 file changed, 11 insertions(+), 26 deletions(-)
Pulling in uncommitted change 175755 at refs/changes/55/175755/1
[unstable 908d4cc3] MB-52378: Add INCLUDE keyword in indexes with missingLeadingKey
 Author: Sai Krishna Teja Kommaraju 
 Date: Mon Jun 6 19:36:28 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 175790 at refs/changes/90/175790/1
Total 19 (delta 16), reused 19 (delta 16)
[unstable 34e7eca1] MB-51825: Add common.GetNumVBuckets function to fetch numVBuckets
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 7 09:10:56 2022 +0530
 1 file changed, 13 insertions(+)
Pulling in uncommitted change 175820 at refs/changes/20/175820/4
Total 48 (delta 39), reused 48 (delta 39)
[unstable 08a0455b] MB-52457 : Provide a global function for query to pass deploymentModel
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 7 18:40:44 2022 +0530
 4 files changed, 18 insertions(+), 6 deletions(-)
Pulling in uncommitted change 175910 at refs/changes/10/175910/1
Total 10 (delta 7), reused 10 (delta 7)
[unstable b61ac0d4] MB-52097 (7.2.0 1312) Accept new clientCertFile, clientKeyFile flags
 Author: cherkauer-couchbase 
 Date: Wed Jun 8 10:12:30 2022 -0700
 3 files changed, 61 insertions(+), 18 deletions(-)
Pulling in uncommitted change 175934 at refs/changes/34/175934/5
Total 11 (delta 9), reused 11 (delta 9)
[unstable 655fb089] MB-51425 Keep minNumShard consistent with max_cpu_percent unless changed
 Author: Varun Velamuri 
 Date: Thu Jun 9 16:03:19 2022 +0530
 2 files changed, 24 insertions(+), 17 deletions(-)
Pulling in uncommitted change 174179 at refs/changes/79/174179/10
Total 8 (delta 1), reused 2 (delta 1)
[unstable c4d455a] MB-51993: Add dynamic resizing for LSS flush buffers
 Author: saptarshi.sen 
 Date: Fri Apr 22 18:10:00 2022 -0700
 6 files changed, 833 insertions(+), 8 deletions(-)
Pulling in uncommitted change 174289 at refs/changes/89/174289/8
Total 12 (delta 4), reused 6 (delta 4)
[unstable ae9ab12] MB-51993: Add LSS flush buffer usage to aggregate stats
 Author: saptarshi.sen 
 Date: Wed Apr 27 18:14:59 2022 -0700
 2 files changed, 18 insertions(+)
Pulling in uncommitted change 175640 at refs/changes/40/175640/10
Total 23 (delta 18), reused 22 (delta 18)
[unstable 0345ba9] MB-51993: AutoTune LSS flush buffers
 Author: saptarshi.sen 
 Date: Tue May 24 10:47:56 2022 -0700
 9 files changed, 1442 insertions(+), 27 deletions(-)
Pulling in uncommitted change 175838 at refs/changes/38/175838/1
Total 6 (delta 4), reused 6 (delta 4)
[unstable 2b44b0f] MB-52283: Add derived bin stats to json jemalloc stats
 Author: akhilmd 
 Date: Wed Jun 8 01:09:53 2022 +0530
 3 files changed, 123 insertions(+), 1 deletion(-)
Building cmakefiles and deps
Building main product

... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo


... A new version of repo (2.21) is available.
... You should upgrade soon:
    cp /opt/build/.repo/repo/repo /home/buildbot/bin/repo

Testing

Started testing at 2022/06/12 10:03:55
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Jun 12 10:05:06 suite_setUp (rebalance.rebalancein.RebalanceInTests) ... ok
Jun 12 10:09:27 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jun 12 10:13:24 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Jun 12 10:14:02 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Jun 12 10:15:15 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Jun 12 10:26:18 test_employee_dataset_startkey_endkey_queries_rebalance_in (view.viewquerytests.ViewQueryTests) ... ok
Jun 12 10:26:53 b" 'stop_on_failure': 'True'}"
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jun 12 10:26:53 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_1'}"
Jun 12 10:26:53 b'-->result: '
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Jun 12 10:26:53 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_2'}"
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Jun 12 10:26:53 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_3'}"
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:26:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Jun 12 10:26:53 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_4'}"
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:26:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:26:53 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'
Jun 12 10:26:53 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_5'}"
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:26:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:26:53 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jun 12 10:26:53 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Jun 12 10:26:53 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Jun 12 10:26:53 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_6'}"
Jun 12 10:26:53 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:26:53 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:26:53 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jun 12 10:26:53 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jun 12 10:26:53 b'testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_6'ok
Jun 12 10:30:21 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Jun 12 10:33:57 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Jun 12 10:36:31 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Jun 12 10:36:32 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Jun 12 10:36:32 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_7'}"
Jun 12 10:36:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:36:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:36:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jun 12 10:36:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jun 12 10:36:32 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Jun 12 10:36:32 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Jun 12 10:36:32 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-04-17/test_8'}"
Jun 12 10:36:32 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Jun 12 10:36:32 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Jun 12 10:36:32 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Jun 12 10:36:32 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Jun 12 10:36:32 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Jun 12 10:36:32 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Jun 12 10:36:32 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jun 12 10:36:32 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Jun 12 10:36:32 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Jun 12 10:36:32 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Jun 12 10:36:32 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Jun 12 10:36:32 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Jun 12 10:36:32 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Jun 12 10:36:32 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.035s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-06-12T10:36:43.847+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 4793 us

2022-06-12T10:36:43.848+05:30 [INFO][FDB] Forestdb opened database file test
2022-06-12T10:36:43.852+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-06-12T10:36:43.852+05:30 [INFO][FDB] Forestdb opened database file test
2022-06-12T10:36:43.855+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 1m59.566191462s -> 83635.68227543784 items/s
Incr build: 10000000 items took 1m1.497552917s -> 162608.09618711937 items/s
Main Index: {
"node_count":             14222114,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       1,
"next_pointers_per_node": 1.3333,
"memory_used":            1339854196,
"node_allocs":            14222114,
"node_frees":             0,
"level_node_distribution":{
"level0": 10667102,
"level1": 2666503,
"level2": 666210,
"level3": 166646,
"level4": 41736,
"level5": 10485,
"level6": 2583,
"level7": 619,
"level8": 181,
"level9": 32,
"level10": 14,
"level11": 2,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (181.07s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.00s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	205.069s
=== RUN   TestConnPoolBasicSanity
2022-06-12T10:40:11.948+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-06-12T10:40:12.154+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:40:12.948+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-06-12T10:40:16.951+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-06-12T10:40:56.705+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:40:56.968+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.76s)
=== RUN   TestLongevity
2022-06-12T10:41:00.707+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-06-12T10:41:39.195+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:41:39.727+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.49s)
=== RUN   TestSustainedHighConns
2022-06-12T10:41:43.197+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Allocating 16 Connections
cp.curActConns = 0
Returning 3 Connections
cp.curActConns = 12
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 4 Connections
cp.curActConns = 13
Allocating 6 Connections
Returning 2 Connections
cp.curActConns = 17
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 23
Returning 1 Connections
cp.curActConns = 25
Returning 3 Connections
Allocating 15 Connections
cp.curActConns = 34
Returning 4 Connections
cp.curActConns = 33
Returning 3 Connections
Allocating 8 Connections
cp.curActConns = 38
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 39
Returning 4 Connections
Allocating 9 Connections
cp.curActConns = 44
Returning 3 Connections
Allocating 7 Connections
cp.curActConns = 46
Returning 1 Connections
cp.curActConns = 47
Allocating 14 Connections
Returning 4 Connections
cp.curActConns = 57
Returning 0 Connections
Allocating 5 Connections
cp.curActConns = 57
Returning 3 Connections
cp.curActConns = 59
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 59
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 60
Returning 0 Connections
Allocating 5 Connections
cp.curActConns = 65
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 68
Returning 3 Connections
cp.curActConns = 77
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 78
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 79
Returning 3 Connections
cp.curActConns = 90
Returning 1 Connections
cp.curActConns = 92
Returning 2 Connections
Allocating 18 Connections
cp.curActConns = 101
Returning 2 Connections
cp.curActConns = 106
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 104
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 112
Returning 0 Connections
cp.curActConns = 122
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 122
Returning 2 Connections
Allocating 8 Connections
cp.curActConns = 124
Returning 1 Connections
cp.curActConns = 127
Returning 4 Connections
Allocating 8 Connections
cp.curActConns = 131
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 135
Returning 2 Connections
cp.curActConns = 142
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 146
Returning 1 Connections
cp.curActConns = 149
Returning 2 Connections
Allocating 15 Connections
cp.curActConns = 157
Returning 3 Connections
cp.curActConns = 159
Returning 2 Connections
Allocating 18 Connections
cp.curActConns = 169
Returning 0 Connections
cp.curActConns = 175
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 172
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 180
Returning 2 Connections
cp.curActConns = 184
Returning 2 Connections
Allocating 10 Connections
cp.curActConns = 192
Returning 0 Connections
Allocating 10 Connections
cp.curActConns = 192
Returning 0 Connections
cp.curActConns = 202
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 203
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 201
Returning 2 Connections
Allocating 9 Connections
cp.curActConns = 203
Returning 2 Connections
cp.curActConns = 206
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 213
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 212
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 207
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 205
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 205
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 207
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 207
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 206
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 204
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 204
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 208
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 209
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 211
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 212
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 212
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 213
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 213
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 211
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 209
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 212
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 0 Connections
cp.curActConns = 215
Allocating 4 Connections
Returning 0 Connections
cp.curActConns = 219
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 214
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 219
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 218
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 217
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 220
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 219
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 218
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 222
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 222
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-06-12T10:42:38.265+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:42:39.231+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.07s)
=== RUN   TestLowWM
2022-06-12T10:42:42.267+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-06-12T10:43:42.283+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-06-12T10:44:42.299+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-06-12T10:44:47.780+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:44:48.301+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN   TestTotalConns
2022-06-12T10:44:51.781+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-06-12T10:45:05.974+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:45:06.789+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.19s)
=== RUN   TestUpdateTickRate
2022-06-12T10:45:09.976+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-06-12T10:45:30.817+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-06-12T10:45:30.986+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.84s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	322.902s
Starting server: attempt 1

Functional tests

2022/06/12 10:47:42 In TestMain()
2022/06/12 10:47:42 otp node fetch error: json: cannot unmarshal string into Go value of type couchbase.Pool
2022/06/12 10:47:42 Initialising services with role: kv,n1ql on node: 127.0.0.1:9000
2022/06/12 10:47:43 Initialising web UI on node: 127.0.0.1:9000
2022/06/12 10:47:43 InitWebCreds, response is: {"newBaseUri":"http://127.0.0.1:9000/"}
2022/06/12 10:47:44 Setting data quota of 1500M and Index quota of 1500M
2022/06/12 10:47:45 Adding node: https://127.0.0.1:19001 with role: kv,index to the cluster
2022/06/12 10:47:52 AddNode: Successfully added node: 127.0.0.1:9001 (role kv,index), response: {"otpNode":"n_1@127.0.0.1"}
2022/06/12 10:47:57 Rebalance progress: 0
2022/06/12 10:48:02 Rebalance progress: 0
2022/06/12 10:48:07 Rebalance progress: 50
2022/06/12 10:48:12 Rebalance progress: 100
2022/06/12 10:48:17 Created bucket default, responseBody: 
2022/06/12 10:48:22 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022/06/12 10:48:22 Successfully initialised cluster
2022/06/12 10:48:22 Cluster status: map[127.0.0.1:9001:[index kv] 172.31.5.112:9000:[kv n1ql]]
2022-06-12T10:48:22.535+05:30 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 127.0.0.1:9106.  Error = EOF. Kill Pipe.
2022-06-12T10:48:22.536+05:30 [Error] FollowerSyncProxy.receiveAndUpdateAcceptedEpoch(): Error encountered = Server Error : SyncProxy.listen(): channel closed. Terminate
2022-06-12T10:48:22.536+05:30 [Error] WatcherServer.runOnce() : Watcher fail to synchronized with peer 127.0.0.1:9106
2022/06/12 10:48:23 ChangeIndexerSettings: Host  Port 0 Nodes []
2022/06/12 10:48:23 Changing config key queryport.client.settings.backfillLimit to value 0
2022/06/12 10:48:23 Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused


goroutine 1 [running]:
panic({0xe67f80, 0xc0005281a0})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/panic.go:941 +0x397 fp=0xc0003afca8 sp=0xc0003afbe8 pc=0x43a6b7
log.Panicf({0x10007f5?, 0x1e?}, {0xc0003afdf8?, 0x27?, 0xe5d640?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/log/log.go:392 +0x67 fp=0xc0003afcf0 sp=0xc0003afca8 pc=0x5bc647
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/functionaltests.TestMain(0x445111?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:104 +0x5c7 fp=0xc0003afec8 sp=0xc0003afcf0 pc=0xd15f07
main.main()
	_testmain.go:477 +0x1d3 fp=0xc0003aff80 sp=0xc0003afec8 pc=0xdeafb3
runtime.main()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:250 +0x212 fp=0xc0003affe0 sp=0xc0003aff80 pc=0x43d232
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0003affe8 sp=0xc0003affe0 pc=0x46db21

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cfb0 sp=0xc00005cf90 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.forcegchelper()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:301 +0xad fp=0xc00005cfe0 sp=0xc00005cfb0 pc=0x43d48d
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46db21
created by runtime.init.6
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:289 +0x25

goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000058790 sp=0xc000058770 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000587c8 sp=0xc000058790 pc=0x428757
runtime.gcenable.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:177 +0x26 fp=0xc0000587e0 sp=0xc0000587c8 pc=0x41e306
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000587e8 sp=0xc0000587e0 pc=0x46db21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:177 +0x6b

goroutine 19 [GC scavenge wait]:
runtime.gopark(0x434ff5d3a4c?, 0x10000?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000058f20 sp=0xc000058f00 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc000058fc8 sp=0xc000058f20 pc=0x426565
runtime.gcenable.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:178 +0x26 fp=0xc000058fe0 sp=0xc000058fc8 pc=0x41e2a6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x46db21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:178 +0xaa

goroutine 3 [finalizer wait]:
runtime.gopark(0x0?, 0x10784b0?, 0x0?, 0xa0?, 0x2000000020?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c630 sp=0xc00005c610 pc=0x43d5f6
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:367
runtime.runfinq()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc00005c7e0 sp=0xc00005c630 pc=0x41d3b3
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46db21
created by runtime.createfing
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mfinal.go:157 +0x45

goroutine 20 [select]:
runtime.gopark(0xc000059798?, 0x2?, 0x0?, 0x0?, 0xc00005978c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000059618 sp=0xc0000595f8 pc=0x43d5f6
runtime.selectgo(0xc000059798, 0xc000059788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000059758 sp=0xc000059618 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc0000c20d8)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:394 +0x67 fp=0xc0000597c8 sp=0xc000059758 pc=0x7842a7
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:556 +0x26 fp=0xc0000597e0 sp=0xc0000597c8 pc=0x784f06
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000597e8 sp=0xc0000597e0 pc=0x46db21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:556 +0x37a

goroutine 21 [select]:
runtime.gopark(0xc000059f98?, 0x2?, 0x0?, 0x0?, 0xc000059f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000059e08 sp=0xc000059de8 pc=0x43d5f6
runtime.selectgo(0xc000059f98, 0xc000059f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000059f48 sp=0xc000059e08 pc=0x44d072
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc0000c20f0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:314 +0x85 fp=0xc000059fc8 sp=0xc000059f48 pc=0x783cc5
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:557 +0x26 fp=0xc000059fe0 sp=0xc000059fc8 pc=0x784ea6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x46db21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:557 +0x3ca

goroutine 22 [IO wait]:
runtime.gopark(0xc0000929c0?, 0xc00004c500?, 0x70?, 0x78?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000497800 sp=0xc0004977e0 pc=0x43d5f6
runtime.netpollblock(0xc0001ad000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000497838 sp=0xc000497800 pc=0x436097
internal/poll.runtime_pollWait(0x7fc8b06a7ff0, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000497858 sp=0xc000497838 pc=0x468109
internal/poll.(*pollDesc).wait(0xc000194180?, 0xc0001ad000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000497880 sp=0xc000497858 pc=0x4a1552
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000194180, {0xc0001ad000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000497900 sp=0xc000497880 pc=0x4a28ba
net.(*netFD).Read(0xc000194180, {0xc0001ad000?, 0xc00009e5a0?, 0xc0004979d0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc000497948 sp=0xc000497900 pc=0x667869
net.(*conn).Read(0xc000188020, {0xc0001ad000?, 0x11?, 0xc000497a68?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc000497990 sp=0xc000497948 pc=0x677ae5
bufio.(*Reader).Read(0xc000180180, {0xc000206001, 0x5ff, 0x452834?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc0004979c8 sp=0xc000497990 pc=0x51f9f4
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x191?, {0xc000206001?, 0x8?, 0x20?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc0004979f8 sp=0xc0004979c8 pc=0x7d4325
encoding/json.(*Decoder).refill(0xc0000bb400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000497a48 sp=0xc0004979f8 pc=0x56423f
encoding/json.(*Decoder).readValue(0xc0000bb400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000497a98 sp=0xc000497a48 pc=0x563e3b
encoding/json.(*Decoder).Decode(0xc0000bb400, {0xe988a0, 0xc0000a23e0})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000497ac8 sp=0xc000497a98 pc=0x563a98
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc0000a23c0, 0xc0000c44a0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc000497b08 sp=0xc000497ac8 pc=0x7d3585
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc0000ae690?, 0x4cc7a8?)
	:1 +0x2a fp=0xc000497b28 sp=0xc000497b08 pc=0x7d60aa
net/rpc.(*Server).readRequestHeader(0xc0000ae690, {0x118ae10, 0xc00009e650})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:587 +0x66 fp=0xc000497bf8 sp=0xc000497b28 pc=0x7d2ca6
net/rpc.(*Server).readRequest(0x0?, {0x118ae10, 0xc00009e650})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:547 +0x3b fp=0xc000497cd0 sp=0xc000497bf8 pc=0x7d27fb
net/rpc.(*Server).ServeCodec(0xc0000ae690, {0x118ae10?, 0xc00009e650})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/rpc/server.go:462 +0x87 fp=0xc000497dc8 sp=0xc000497cd0 pc=0x7d1f27
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc00005a760?, 0xc00006bfa0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc000497f38 sp=0xc000497dc8 pc=0x7d4a79
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0x1181820?, 0xc000184030?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc000497f70 sp=0xc000497f38 pc=0x7d5178
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc0000ac270)
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc000497fc0 sp=0xc000497f70 pc=0x7de97d
github.com/couchbase/cbauth.startDefault.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc000497fe0 sp=0xc000497fc0 pc=0x7dec65
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000497fe8 sp=0xc000497fe0 pc=0x46db21
created by github.com/couchbase/cbauth.startDefault
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9

goroutine 29 [chan receive]:
runtime.gopark(0x3?, 0xc0000d8120?, 0x20?, 0x96?, 0xc000539760?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000539678 sp=0xc000539658 pc=0x43d5f6
runtime.chanrecv(0xc0000a28a0, 0xc000539790, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:577 +0x56c fp=0xc000539708 sp=0xc000539678 pc=0x40b5ac
runtime.chanrecv2(0xc0000aa670?, 0x16?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:445 +0x18 fp=0xc000539730 sp=0xc000539708 pc=0x40b018
github.com/couchbase/goutils/systemeventlog.(*SystemEventLoggerImpl).logEvents(0xc0001600e0)
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:186 +0xb7 fp=0xc0005397c8 sp=0xc000539730 pc=0xb71ff7
github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger.func1()
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x26 fp=0xc0005397e0 sp=0xc0005397c8 pc=0xb71906
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005397e8 sp=0xc0005397e0 pc=0x46db21
created by github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x1d6

goroutine 40 [select]:
runtime.gopark(0xc0004aaf90?, 0x2?, 0xd8?, 0xad?, 0xc0004aaf24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004aad90 sp=0xc0004aad70 pc=0x43d5f6
runtime.selectgo(0xc0004aaf90, 0xc0004aaf20, 0xc00018e540?, 0x0, 0xc000025e30?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004aaed0 sp=0xc0004aad90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc000478000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc0004aafc8 sp=0xc0004aaed0 pc=0x76b655
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc0004aafe0 sp=0xc0004aafc8 pc=0x7680e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004aafe8 sp=0xc0004aafe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 39 [select]:
runtime.gopark(0xc0004a9f68?, 0x4?, 0x3?, 0x0?, 0xc0004a9db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004a9c00 sp=0xc0004a9be0 pc=0x43d5f6
runtime.selectgo(0xc0004a9f68, 0xc0004a9da8, 0xc00018f440?, 0x0, 0xc00005a501?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004a9d40 sp=0xc0004a9c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc000478000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc0004a9fc8 sp=0xc0004a9d40 pc=0x76a565
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc0004a9fe0 sp=0xc0004a9fc8 pc=0x768146
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004a9fe8 sp=0xc0004a9fe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 48 [select]:
runtime.gopark(0xc00045df38?, 0x2?, 0x0?, 0x30?, 0xc00045df24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00045dd58 sp=0xc00045dd38 pc=0x43d5f6
runtime.selectgo(0xc00045df38, 0xc00045df20, 0xc0005521c0?, 0x0, 0xc00050d350?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00045de98 sp=0xc00045dd58 pc=0x44d072
github.com/couchbase/gometa/protocol.RunWatcherServerWithRequest2({0xc00003ac50, 0xe}, {0x1184218, 0xc000548300}, {0x11908b8, 0xc000548300}, {0x118fda0, 0x1a5a0f8}, 0xc000552150, 0xc000042a20, ...)
	/opt/build/goproj/src/github.com/couchbase/gometa/protocol/watcherServer.go:72 +0x20e fp=0xc00045df68 sp=0xc00045de98 pc=0xc17cee
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4082 +0x9f fp=0xc00045dfe0 sp=0xc00045df68 pc=0xc76f1f
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00045dfe8 sp=0xc00045dfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).startWatcher
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4082 +0x1f8

goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053a758 sp=0xc00053a738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00053a7e0 sp=0xc00053a758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053a7e8 sp=0xc00053a7e0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x434ff3dbec5?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005df58 sp=0xc00005df38 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005dfe0 sp=0xc00005df58 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 51 [GC worker (idle)]:
runtime.gopark(0x43e4333ab8a?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053af58 sp=0xc00053af38 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00053afe0 sp=0xc00053af58 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053afe8 sp=0xc00053afe0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x43e4333632a?, 0x1?, 0x5e?, 0x63?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053b758 sp=0xc00053b738 pc=0x43d5f6
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00053b7e0 sp=0xc00053b758 pc=0x4203e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053b7e8 sp=0xc00053b7e0 pc=0x46db21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/mgc.go:1131 +0x25

goroutine 9 [select]:
runtime.gopark(0xc000686eb8?, 0x2?, 0x0?, 0x0?, 0xc000686e90?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000686d18 sp=0xc000686cf8 pc=0x43d5f6
runtime.selectgo(0xc000686eb8, 0xc000686e8c, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000686e58 sp=0xc000686d18 pc=0x44d072
github.com/couchbase/indexing/secondary/manager/client.(*watcher).waitForReady(0xc000548300, 0xc000042a20, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:5749 +0x239 fp=0xc000686f20 sp=0xc000686e58 pc=0xc81e39
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).retryHelper(0xc00012e4d0, 0x0?, 0x0?, {0xc00003ac50, 0xe}, {0xc00048fce0, 0x1b}, 0x0?, 0xc000187ea0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:4016 +0x65 fp=0xc000686f88 sp=0xc000686f20 pc=0xc76725
github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:337 +0x46 fp=0xc000686fe0 sp=0xc000686f88 pc=0xc5fd86
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000686fe8 sp=0xc000686fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/client.(*MetadataProvider).WatchMetadata
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/client/metadata_provider.go:337 +0x4d4

goroutine 89 [runnable]:
runtime.gopark(0xc00045aee0?, 0x6?, 0xa0?, 0xab?, 0xc00045acf4?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00045ab58 sp=0xc00045ab38 pc=0x43d5f6
runtime.selectgo(0xc00045aee0, 0xc00045ace8, 0x1007b4b?, 0x0, 0xc00045acd0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00045ac98 sp=0xc00045ab58 pc=0x44d072
net/http.(*persistConn).roundTrip(0xc000026240, 0xc0004665c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2620 +0x974 fp=0xc00045af50 sp=0xc00045ac98 pc=0x76c514
net/http.(*Transport).roundTrip(0x19a2920, 0xc000515500)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:594 +0x7c9 fp=0xc00045b188 sp=0xc00045af50 pc=0x760009
net/http.(*Transport).RoundTrip(0xc000515500?, 0x1180fa0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc00045b1a8 sp=0xc00045b188 pc=0x7482b9
net/http.send(0xc000515400, {0x1180fa0, 0x19a2920}, {0xf8d300?, 0x48e801?, 0x1a23aa0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:252 +0x5d8 fp=0xc00045b388 sp=0xc00045b1a8 pc=0x709bd8
net/http.(*Client).send(0xc000182c60, 0xc000515400, {0x7fc8b06f5c30?, 0x150?, 0x1a23aa0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:176 +0x9b fp=0xc00045b400 sp=0xc00045b388 pc=0x70947b
net/http.(*Client).do(0xc000182c60, 0xc000515400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:725 +0x8f5 fp=0xc00045b600 sp=0xc00045b400 pc=0x70b8b5
net/http.(*Client).Do(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc0000390e0?, 0x1b?}, 0xc00045b998, {0x0, 0x0}, 0x0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc00045b700 sp=0xc00045b600 pc=0x7e3c69
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc0006682d0, {0xfffd59, 0x6}, {0xeeb2c0?, 0x118ac01?}, {0xe3f2a0, 0xc000195318}, 0xc000043620?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc00045b948 sp=0xc00045b700 pc=0x806553
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth({0xc000043620, 0x3f}, {0x1180920?, 0xc000048a80})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:594 +0x125 fp=0xc00045b9c0 sp=0xc00045b948 pc=0x808505
github.com/couchbase/indexing/secondary/dcp.Connect({0xc000043620, 0x3f})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:600 +0xce fp=0xc00045bb58 sp=0xc00045b9c0 pc=0x8086ae
github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier({0xc000043620, 0x3f}, {0x1000e15, 0x7})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:236 +0x18b fp=0xc00045bda0 sp=0xc00045bb58 pc=0xada26b
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).watchClusterChanges(0xc00012e420)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2047 +0xf2 fp=0xc00045bfc8 sp=0xc00045bda0 pc=0xcb3392
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x26 fp=0xc00045bfe0 sp=0xc00045bfc8 pc=0xca6fa6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00045bfe8 sp=0xc00045bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x5c5

goroutine 91 [select]:
runtime.gopark(0xc000539f90?, 0x2?, 0xd0?, 0xb7?, 0xc000539f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000539e18 sp=0xc000539df8 pc=0x43d5f6
runtime.selectgo(0xc000539f90, 0xc000539f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000539f58 sp=0xc000539e18 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).listenMetaChange(0xc000194b00, 0xc00021e2a0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1825 +0x70 fp=0xc000539fc0 sp=0xc000539f58 pc=0xca2230
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1818 +0x2a fp=0xc000539fe0 sp=0xc000539fc0 pc=0xca218a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1818 +0x28a

goroutine 32 [select]:
runtime.gopark(0xc000536798?, 0x2?, 0x40?, 0x0?, 0xc00053678c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc000536618 sp=0xc0005365f8 pc=0x43d5f6
runtime.selectgo(0xc000536798, 0xc000536788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc000536758 sp=0xc000536618 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*schedTokenMonitor).updater(0xc0000ae280)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2365 +0x92 fp=0xc0005367c8 sp=0xc000536758 pc=0xcb5692
github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2162 +0x26 fp=0xc0005367e0 sp=0xc0005367c8 pc=0xcb3fc6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0005367e8 sp=0xc0005367e0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2162 +0x2e5

goroutine 77 [select]:
runtime.gopark(0xc00006bf90?, 0x2?, 0xd8?, 0xbd?, 0xc00006bf24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006bd90 sp=0xc00006bd70 pc=0x43d5f6
runtime.selectgo(0xc00006bf90, 0xc00006bf20, 0xc0000a0640?, 0x0, 0xc000183380?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00006bed0 sp=0xc00006bd90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc000026240)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00006bfc8 sp=0xc00006bed0 pc=0x76b655
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x7680e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 90 [chan receive]:
runtime.gopark(0xc00053bed8?, 0x44319b?, 0x20?, 0xbf?, 0x459d45?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053bec8 sp=0xc00053bea8 pc=0x43d5f6
runtime.chanrecv(0xc000519140, 0x0, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:577 +0x56c fp=0xc00053bf58 sp=0xc00053bec8 pc=0x40b5ac
runtime.chanrecv1(0xdf8475800?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/chan.go:440 +0x18 fp=0xc00053bf80 sp=0xc00053bf58 pc=0x40afd8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).logstats(0xc00012e420)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:1386 +0x79 fp=0xc00053bfc8 sp=0xc00053bf80 pc=0xcae139
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x26 fp=0xc00053bfe0 sp=0xc00053bfc8 pc=0xca6f46
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x605

goroutine 76 [select]:
runtime.gopark(0xc0004a8f68?, 0x4?, 0x3?, 0x0?, 0xc0004a8db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004a8c00 sp=0xc0004a8be0 pc=0x43d5f6
runtime.selectgo(0xc0004a8f68, 0xc0004a8da8, 0xc000466600?, 0x0, 0xc00053dd01?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc0004a8d40 sp=0xc0004a8c00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc000026240)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc0004a8fc8 sp=0xc0004a8d40 pc=0x76a565
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc0004a8fe0 sp=0xc0004a8fc8 pc=0x768146
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004a8fe8 sp=0xc0004a8fe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 69 [select]:
runtime.gopark(0xc00001bba0?, 0x3?, 0x0?, 0x30?, 0xc00001bb1a?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00001b978 sp=0xc00001b958 pc=0x43d5f6
runtime.selectgo(0xc00001bba0, 0xc00001bb14, 0x3?, 0x0, 0x412065?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00001bab8 sp=0xc00001b978 pc=0x44d072
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc000186310?, {0x1015ed7, 0x1b}, 0xc00001be68, 0xc0000ce540)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:301 +0x429 fp=0xc00001be40 sp=0xc00001bab8 pc=0x9c1429
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:259
github.com/couchbase/cbauth/metakv.RunObserveChildren({0x1015ed7?, 0x0?}, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x58 fp=0xc00001be88 sp=0xc00001be40 pc=0x9c19d8
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0?, {0x0?, 0x0?})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1576 +0xc7 fp=0xc00001bf00 sp=0xc00001be88 pc=0xb76f87
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc00001bfa0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc00001bf38 sp=0xc00001bf00 pc=0xad7a83
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1581 +0xdf fp=0xc00001bfe0 sp=0xc00001bf38 pc=0xb76e3f
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00001bfe8 sp=0xc00001bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1569 +0xaf

goroutine 87 [IO wait]:
runtime.gopark(0xc0003c56c0?, 0xc000050f00?, 0x60?, 0xb7?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00049b6f0 sp=0xc00049b6d0 pc=0x43d5f6
runtime.netpollblock(0xc0001ac000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00049b728 sp=0xc00049b6f0 pc=0x436097
internal/poll.runtime_pollWait(0x7fc8b06a7f00, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00049b748 sp=0xc00049b728 pc=0x468109
internal/poll.(*pollDesc).wait(0xc000194800?, 0xc0001ac000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00049b770 sp=0xc00049b748 pc=0x4a1552
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000194800, {0xc0001ac000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00049b7f0 sp=0xc00049b770 pc=0x4a28ba
net.(*netFD).Read(0xc000194800, {0xc0001ac000?, 0xc00049b8b8?, 0x551e45?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc00049b838 sp=0xc00049b7f0 pc=0x667869
net.(*conn).Read(0xc000188368, {0xc0001ac000?, 0x198?, 0xe53ee0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc00049b880 sp=0xc00049b838 pc=0x677ae5
net/http.(*persistConn).Read(0xc000478000, {0xc0001ac000?, 0xc00049b910?, 0xc00013c120?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1929 +0x4e fp=0xc00049b8e0 sp=0xc00049b880 pc=0x768b4e
bufio.(*Reader).fill(0xc0003c2c60)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc00049b918 sp=0xc00049b8e0 pc=0x51f423
bufio.(*Reader).ReadSlice(0xc0003c2c60, 0x5a?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc00049b968 sp=0xc00049b918 pc=0x52000f
net/http/internal.readChunkLine(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc00049b9b8 sp=0xc00049b968 pc=0x706a85
net/http/internal.(*chunkedReader).beginChunk(0xc0005068d0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc00049b9e8 sp=0xc00049b9b8 pc=0x706508
net/http/internal.(*chunkedReader).Read(0xc0005068d0, {0xc000162000?, 0x0?, 0xf?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc00049ba68 sp=0xc00049b9e8 pc=0x7067ce
net/http.(*body).readLocked(0xc00018f440, {0xc000162000?, 0x1000?, 0x1?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:844 +0x3c fp=0xc00049bab8 sp=0xc00049ba68 pc=0x75d77c
net/http.(*body).Read(0xc0004aee21?, {0xc000162000?, 0xc000024cc0?, 0xc00020a000?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:836 +0x125 fp=0xc00049bb30 sp=0xc00049bab8 pc=0x75d645
net/http.(*bodyEOFSignal).Read(0xc00018f480, {0xc000162000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2774 +0x142 fp=0xc00049bbb0 sp=0xc00049bb30 pc=0x76d282
bufio.(*Reader).fill(0xc00049bf58)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc00049bbe8 sp=0xc00049bbb0 pc=0x51f423
bufio.(*Reader).ReadSlice(0xc00049bf58, 0x65?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc00049bc38 sp=0xc00049bbe8 pc=0x52000f
bufio.(*Reader).collectFragments(0xf6cb00?, 0x98?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:446 +0x74 fp=0xc00049bce8 sp=0xc00049bc38 pc=0x520454
bufio.(*Reader).ReadBytes(0xc000518b40?, 0x5c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:474 +0x1d fp=0xc00049bd68 sp=0xc00049bce8 pc=0x5205fd
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).waitForChange(0xc000038e70?, 0x2c?, 0x102fa05?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:620 +0x4b fp=0xc00049be18 sp=0xc00049bd68 pc=0xad454b
github.com/couchbase/indexing/secondary/common.(*internalVersionMonitor).notifier(0xc0001b8a10)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:709 +0x493 fp=0xc00049bfc8 sp=0xc00049be18 pc=0xad4df3
github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x26 fp=0xc00049bfe0 sp=0xc00049bfc8 pc=0xad0766
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/common.newInternalVersionMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:232 +0x336

goroutine 72 [select]:
runtime.gopark(0xc00006af68?, 0x4?, 0x3?, 0x0?, 0xc00006adb0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006ac00 sp=0xc00006abe0 pc=0x43d5f6
runtime.selectgo(0xc00006af68, 0xc00006ada8, 0xc000151f40?, 0x0, 0xc000536d01?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00006ad40 sp=0xc00006ac00 pc=0x44d072
net/http.(*persistConn).readLoop(0xc000026360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2213 +0xda5 fp=0xc00006afc8 sp=0xc00006ad40 pc=0x76a565
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x26 fp=0xc00006afe0 sp=0xc00006afc8 pc=0x768146
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1750 +0x173e

goroutine 73 [select]:
runtime.gopark(0xc00006ff90?, 0x2?, 0xd8?, 0xfd?, 0xc00006ff24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006fd90 sp=0xc00006fd70 pc=0x43d5f6
runtime.selectgo(0xc00006ff90, 0xc00006ff20, 0xc0000a05c0?, 0x0, 0xc000468f60?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00006fed0 sp=0xc00006fd90 pc=0x44d072
net/http.(*persistConn).writeLoop(0xc000026360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00006ffc8 sp=0xc00006fed0 pc=0x76b655
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x26 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x7680e6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46db21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1751 +0x1791

goroutine 85 [IO wait]:
runtime.gopark(0xc000003860?, 0xc000050f00?, 0x10?, 0xba?, 0x483442?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc0004ab9a0 sp=0xc0004ab980 pc=0x43d5f6
runtime.netpollblock(0xc000020000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0004ab9d8 sp=0xc0004ab9a0 pc=0x436097
internal/poll.runtime_pollWait(0x7fc8b06a7c30, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0004ab9f8 sp=0xc0004ab9d8 pc=0x468109
internal/poll.(*pollDesc).wait(0xc0000d6600?, 0xc000020000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc0004aba20 sp=0xc0004ab9f8 pc=0x4a1552
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0000d6600, {0xc000020000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0004abaa0 sp=0xc0004aba20 pc=0x4a28ba
net.(*netFD).Read(0xc0000d6600, {0xc000020000?, 0xf57ae0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004abae8 sp=0xc0004abaa0 pc=0x667869
net.(*conn).Read(0xc0000a81f0, {0xc000020000?, 0xc00050d360?, 0xe?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/net.go:183 +0x45 fp=0xc0004abb30 sp=0xc0004abae8 pc=0x677ae5
net/http.(*persistConn).Read(0xc000026360, {0xc000020000?, 0x1000?, 0x1000?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:1929 +0x4e fp=0xc0004abb90 sp=0xc0004abb30 pc=0x768b4e
bufio.(*Reader).fill(0xc0000a35c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:106 +0x103 fp=0xc0004abbc8 sp=0xc0004abb90 pc=0x51f423
bufio.(*Reader).ReadSlice(0xc0000a35c0, 0x88?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/bufio/bufio.go:371 +0x2f fp=0xc0004abc18 sp=0xc0004abbc8 pc=0x52000f
net/http/internal.readChunkLine(0xef7ac0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc0004abc68 sp=0xc0004abc18 pc=0x706a85
net/http/internal.(*chunkedReader).beginChunk(0xc0000254d0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc0004abc98 sp=0xc0004abc68 pc=0x706508
net/http/internal.(*chunkedReader).Read(0xc0000254d0, {0xc00050b400?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc0004abd18 sp=0xc0004abc98 pc=0x7067ce
net/http.(*body).readLocked(0xc000151f40, {0xc00050b400?, 0xc00050d400?, 0xc00005e5b8?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:844 +0x3c fp=0xc0004abd68 sp=0xc0004abd18 pc=0x75d77c
net/http.(*body).Read(0x1000000000000?, {0xc00050b400?, 0x0?, 0x7fc8d851af18?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transfer.go:836 +0x125 fp=0xc0004abde0 sp=0xc0004abd68 pc=0x75d645
net/http.(*bodyEOFSignal).Read(0xc000151f80, {0xc00050b400, 0x200, 0x200})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/net/http/transport.go:2774 +0x142 fp=0xc0004abe60 sp=0xc0004abde0 pc=0x76d282
encoding/json.(*Decoder).refill(0xc00011e780)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:165 +0x17f fp=0xc0004abeb0 sp=0xc0004abe60 pc=0x56423f
encoding/json.(*Decoder).readValue(0xc00011e780)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:140 +0xbb fp=0xc0004abf00 sp=0xc0004abeb0 pc=0x563e3b
encoding/json.(*Decoder).Decode(0xc00011e780, {0xe3dae0, 0xc00007e5f0})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/encoding/json/stream.go:63 +0x78 fp=0xc0004abf30 sp=0xc0004abf00 pc=0x563a98
github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x10b fp=0xc0004abfe0 sp=0xc0004abf30 pc=0x9c18cb
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0004abfe8 sp=0xc0004abfe0 pc=0x46db21
created by github.com/couchbase/cbauth/metakv.doRunObserveChildren
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:280 +0x2eb

goroutine 92 [select]:
runtime.gopark(0xc00053c750?, 0x2?, 0x0?, 0x30?, 0xc00053c714?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/proc.go:361 +0xd6 fp=0xc00053c590 sp=0xc00053c570 pc=0x43d5f6
runtime.selectgo(0xc00053c750, 0xc00053c710, 0x103dc91?, 0x0, 0xe67480?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/select.go:328 +0x772 fp=0xc00053c6d0 sp=0xc00053c590 pc=0x44d072
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).logstats(0xc000194b00, 0xc00021e2a0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1844 +0x238 fp=0xc00053c7c0 sp=0xc00053c6d0 pc=0xca24b8
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1819 +0x2a fp=0xc00053c7e0 sp=0xc00053c7c0 pc=0xca212a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.1/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00053c7e8 sp=0xc00053c7e0 pc=0x46db21
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1819 +0x2ed
signal: aborted (core dumped)
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests	40.887s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 53006    0 53006    0     0  2530k      0 --:--:-- --:--:-- --:--:-- 2588k
2022/06/12 10:48:25 In TestMain()
2022/06/12 10:48:26 Changing config key indexer.api.enableTestServer to value true
2022/06/12 10:48:26 Using forestdb for creating indexes
2022/06/12 10:48:26 Changing config key indexer.settings.storage_mode to value forestdb
=== RUN   TestRangeWithConcurrentAddMuts
2022/06/12 10:48:31 In TestRangeWithConcurrentAddMuts()
2022/06/12 10:48:31 In DropAllSecondaryIndexes()
2022/06/12 10:48:31 Generating JSON docs
2022/06/12 10:48:31 Setting initial JSON docs in KV
2022/06/12 10:48:31 All indexers are active
2022/06/12 10:48:31 Creating a 2i
2022-06-12T10:48:31.713+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:48:41.715+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestRangeWithConcurrentAddMuts (10.67s)
=== RUN   TestRangeWithConcurrentDelMuts
2022/06/12 10:48:41 In TestRangeWithConcurrentDelMuts()
2022/06/12 10:48:41 Generating JSON docs
2022/06/12 10:48:43 Setting initial JSON docs in KV
2022/06/12 10:48:55 All indexers are active
2022/06/12 10:48:55 Creating a 2i
2022-06-12T10:48:56.069+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:49:06.072+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestRangeWithConcurrentDelMuts (24.36s)
=== RUN   TestScanWithConcurrentIndexOps
2022/06/12 10:49:06 In TestScanWithConcurrentIndexOps()
2022/06/12 10:49:06 Generating JSON docs
2022/06/12 10:49:11 Setting initial JSON docs in KV
2022/06/12 10:49:55 All indexers are active
2022/06/12 10:49:55 Creating a 2i
2022-06-12T10:49:55.970+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:50:05.975+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestScanWithConcurrentIndexOps (59.90s)
=== RUN   TestConcurrentScans_SameIndex
2022/06/12 10:50:05 In TestConcurrentScans_SameIndex()
2022/06/12 10:50:05 Generating JSON docs
2022/06/12 10:50:12 Setting initial JSON docs in KV
2022/06/12 10:50:55 All indexers are active
2022/06/12 10:50:55 Creating a 2i
2022-06-12T10:50:55.532+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:51:05.536+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestConcurrentScans_SameIndex (59.56s)
=== RUN   TestConcurrentScans_MultipleIndexes
2022/06/12 10:51:05 In TestConcurrentScans_MultipleIndexes()
2022/06/12 10:51:05 Generating JSON docs
2022/06/12 10:51:12 Setting initial JSON docs in KV
2022/06/12 10:51:56 All indexers are active
2022/06/12 10:51:56 Creating multiple indexes
2022-06-12T10:51:56.513+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:52:06.516+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestConcurrentScans_MultipleIndexes (60.98s)
=== RUN   TestMutationsWithMultipleIndexBuilds
2022/06/12 10:52:06 In TestMutationsWithMultipleIndexBuilds()
2022/06/12 10:52:06 In DropAllSecondaryIndexes()
2022/06/12 10:52:06 Generating JSON docs
2022/06/12 10:52:13 Setting initial JSON docs in KV
2022-06-12T10:53:01.139+05:30 [Error] Encountered error during create index.  Error: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
2022-06-12T10:53:11.142+05:30 [Error] Fail to create index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
    common_test.go:70: Error in creating the index: Cannot Create Index with Using forestdb. Indexer Storage Mode plasma
        
--- FAIL: TestMutationsWithMultipleIndexBuilds (64.63s)
FAIL
exit status 1
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests	285.277s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 55492    0 55492    0     0   9.9M      0 --:--:-- --:--:-- --:--:-- 10.5M

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[  0%] Built target ns_ui_build_prepare
[  0%] Building Go Modules target ns_minify_js using Go 1.16.6
[  0%] Built target ns_minify_js
[  0%] Building Go Modules target ns_minify_css using Go 1.16.6
[  0%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> chronicle (compile)
[100%] Built target chronicle
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target golang_tls_test using Go 1.17.4
[100%] Built target golang_tls_test
[100%] Building Go target ns_goport using Go 1.18.1
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.1
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.1
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.1
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.1
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:root:__main__
INFO:__main__:TestRunner: parsing args...
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-06-12 10:53:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:53:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:53:52 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:53:52 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/nodes/self error [Errno 111] Connection refused 
2022-06-12 10:53:55 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/nodes/self error [Errno 111] Connection refused 
2022-06-12 10:54:01 | ERROR | MainProcess | MainThread | [rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/nodes/self error [Errno 111] Connection refused 
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [rest_client.get_nodes_version] Node version in cluster 7.2.0-1350-rel-enterprise
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-06-12 10:54:13 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:13 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-06-12 10:54:13 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-06-12 10:54:19 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-06-12 10:54:19 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:19 | ERROR | MainProcess | MainThread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-06-12 10:54:19 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '21', 'memoryTotal': 15466930176, 'memoryFree': 4973436928, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:54:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:54:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 4805693440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:54:20 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:54:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 4687544320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:54:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:54:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '17', 'memoryTotal': 15466930176, 'memoryFree': 4582514688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:54:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:54:22 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:54:22 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:54:22 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-06-12 10:54:22 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-06-12 10:54:22 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-06-12 10:54:22 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:54:27 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:54:28 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-06-12 10:54:28 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-06-12 10:54:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.39977196690008, 'mem_free': 13506842624, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 39.64898729324341
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-06-12 10:54:32 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-06-12 10:54:37 | INFO | MainProcess | MainThread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-06-12 10:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:54:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:54:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:54:38 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-06-12 10:54:38 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:54:40 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-06-12 10:54:40 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:54:40 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-06-12 10:54:40 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 95.84104ms
2022-06-12 10:55:10 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '007b962c-c6b9-435b-9b65-f7958c52d89e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '95.84104ms', 'executionTime': '95.588369ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-06-12 10:55:10 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 444.548205ms
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.862269ms
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.403747ms
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 41.640981ms
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.32597368246049, 'mem_free': 13486829568, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:13 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:16 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-06-12 10:55:16 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-06-12 10:55:17 | ERROR | MainProcess | MainThread | [rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-06-12 10:55:17 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.182s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-06-12 10:55:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 7.2.0-1350-rel-enterprise
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.16491897378356, 'mem_free': 13460586496, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-06-12 10:55:22 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-06-12 10:55:22 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '87', 'memoryTotal': 15466930176, 'memoryFree': 13460586496, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-06-12 10:55:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:55:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '82', 'memoryTotal': 15466930176, 'memoryFree': 13437116416, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:55:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '82', 'memoryTotal': 15466930176, 'memoryFree': 13436538880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:55:23 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '83', 'memoryTotal': 15466930176, 'memoryFree': 13462876160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] --> status:True
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:23 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-06-12 10:55:24 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-06-12 10:55:24 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-06-12 10:55:24 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-06-12 10:55:24 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-06-12 10:55:29 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-06-12 10:55:30 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-06-12 10:55:30 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket default
2022-06-12 10:55:30 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-06-12 10:55:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:55:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:55:31 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:31 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.26031145067361, 'mem_free': 13460279296, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 17.574772596359253
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-06-12 10:55:35 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-06-12 10:55:40 | INFO | MainProcess | test_thread | [rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-06-12 10:55:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:55:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:55:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:55:41 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-06-12 10:55:41 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-06-12 10:55:43 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-06-12 10:55:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:55:43 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-06-12 10:55:43 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.34831ms
2022-06-12 10:56:13 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'c180a5c2-dd48-4f89-b6f3-68cdb234ed9f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.34831ms', 'executionTime': '20.193855ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-06-12 10:56:13 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 391.989961ms
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.699313ms
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-06-12 10:56:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:56:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeeeb95670d6a504a218a399a34e065476djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-06-12 10:56:15 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeeeb95670d6a504a218a399a34e065476djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-06-12 10:56:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 48.81315ms
2022-06-12 10:56:15 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeeeb95670d6a504a218a399a34e065476djob_title) USING GSI
2022-06-12 10:56:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeeeb95670d6a504a218a399a34e065476djob_title) USING GSI
2022-06-12 10:56:16 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeeeb95670d6a504a218a399a34e065476djob_title%29+USING+GSI
2022-06-12 10:56:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 27.85916ms
2022-06-12 10:56:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeeb95670d6a504a218a399a34e065476djob_title'
2022-06-12 10:56:17 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeb95670d6a504a218a399a34e065476djob_title%27
2022-06-12 10:56:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.993647ms
2022-06-12 10:56:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeeb95670d6a504a218a399a34e065476djob_title'
2022-06-12 10:56:18 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeb95670d6a504a218a399a34e065476djob_title%27
2022-06-12 10:56:18 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.358393ms
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.063175ms
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '21397c38-2fc5-4d08-8a38-e942b83b0726', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeeb95670d6a504a218a399a34e065476djob_title', 'index_id': '8b1da1f77d454caa', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.063175ms', 'executionTime': '2.966462ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:19 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-06-12 10:56:19 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 75.271825ms
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-06-12 10:56:20 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-06-12 10:56:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:21 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeeb95670d6a504a218a399a34e065476djob_title'
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeb95670d6a504a218a399a34e065476djob_title%27
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 10.243844ms
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeeeb95670d6a504a218a399a34e065476djob_title ON default USING GSI
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=DROP+INDEX+employeeeb95670d6a504a218a399a34e065476djob_title+ON+default+USING+GSI
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 38.626593ms
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeeb95670d6a504a218a399a34e065476djob_title'
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeb95670d6a504a218a399a34e065476djob_title%27
2022-06-12 10:56:22 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.032169ms
2022-06-12 10:56:22 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'd2539a31-355d-4ecd-a5ff-3c0bf019e3c7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '2.032169ms', 'executionTime': '1.897098ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-06-12 10:56:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:56:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.625284ms
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 39.721372ms
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.45727700939066, 'mem_free': 13342425088, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-06-12 10:56:24 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-06-12 10:56:28 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [rest_client.is_ns_server_running] -->is_ns_server_running?
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-06-12 10:56:28 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 70.904s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_2'}
[2022-06-12 10:56:28,768] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:28,768] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:29,011] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:29,019] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:56:29,028] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 10:56:29,028] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-06-12 10:56:29,029] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 10:56:29,771] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 10:56:29,774] - [task:159] INFO -  {'uptime': '153', 'memoryTotal': 15466930176, 'memoryFree': 13380206592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:56:29,779] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 10:56:29,779] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 10:56:29,779] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 10:56:29,791] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 10:56:29,799] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 10:56:29,799] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 10:56:29,802] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 10:56:29,803] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 10:56:29,803] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 10:56:29,804] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 10:56:29,807] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:56:29,810] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:29,811] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:30,049] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:30,050] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:56:30,091] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:56:30,092] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:56:30,097] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:56:30,097] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:56:30,100] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:56:30,111] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 10:56:30,114] - [task:159] INFO -  {'uptime': '152', 'memoryTotal': 15466930176, 'memoryFree': 13329190912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:56:30,119] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:56:30,123] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 10:56:30,123] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 10:56:30,126] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:56:30,130] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:30,131] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:30,373] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:30,374] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:56:30,414] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:56:30,415] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:56:30,419] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:56:30,420] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:56:30,422] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:56:30,434] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 10:56:30,437] - [task:159] INFO -  {'uptime': '147', 'memoryTotal': 15466930176, 'memoryFree': 13330694144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:56:30,442] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:56:30,446] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 10:56:30,446] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 10:56:30,449] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:56:30,454] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:30,454] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:30,687] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:30,688] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:56:30,729] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:56:30,730] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:56:30,735] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:56:30,736] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:56:30,738] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:56:30,749] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 10:56:30,752] - [task:159] INFO -  {'uptime': '148', 'memoryTotal': 15466930176, 'memoryFree': 13351235584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:56:30,756] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:56:30,760] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 10:56:30,761] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 10:56:30,765] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:56:30,769] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:30,769] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:31,006] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:31,007] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:56:31,047] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:56:31,048] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:56:31,054] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:56:31,054] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:56:31,057] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:56:31,068] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 10:56:31,205] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:56:36,210] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 10:56:36,222] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 10:56:36,227] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:36,228] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:36,461] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:36,462] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:56:36,501] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:56:36,502] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:56:36,503] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 10:56:37,087] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 10:56:37,117] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 10:56:37,117] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 10:56:37,687] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:56:37,775] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:56:37,820] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 10:56:37,822] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:56:37,830] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:37,830] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:38,067] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:38,072] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:38,072] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:38,286] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:38,290] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:38,290] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:38,504] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:38,509] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:38,509] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:38,729] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:42,134] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:56:42,134] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.02247042926428, 'mem_free': 13374259200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:56:42,134] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:56:42,135] - [basetestcase:563] INFO - Time to execute basesetup : 13.36945104598999
[2022-06-12 10:56:42,145] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:56:42,145] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:56:42,152] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:56:42,153] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:56:42,160] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:56:42,160] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:56:42,167] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:56:42,167] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:56:42,172] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:56:42,194] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 10:56:42,195] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 10:56:42,195] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:56:47,207] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 10:56:47,210] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:56:47,210] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:56:47,426] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:56:48,338] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 10:56:48,366] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:56:50,542] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:56:50,558] - [newtuq:82] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-06-12 10:56:51,151] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:56:51,151] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 10:56:51,151] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 10:57:21,180] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 10:57:21,184] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:57:21,188] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:57:21,210] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.153133ms
[2022-06-12 10:57:21,210] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9f0852ef-97b2-4f2a-8029-49d108eec5ae', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.153133ms', 'executionTime': '20.071651ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:57:21,210] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 10:57:21,214] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 10:57:21,218] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 10:57:21,625] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 404.714165ms
[2022-06-12 10:57:21,625] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 10:57:21,636] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:57:21,643] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:57:21,654] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.698091ms
[2022-06-12 10:57:21,677] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:21,678] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:22,033] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:22,051] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:22,054] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 10:57:22,069] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 10:57:22,069] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 10:57:22,079] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 10:57:22,089] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:23,051] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee778701d1ea8c43aeb28e5ede95ebecb1job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 10:57:23,055] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employee778701d1ea8c43aeb28e5ede95ebecb1job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 10:57:23,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.980627ms
[2022-06-12 10:57:23,121] - [base_gsi:282] INFO - BUILD INDEX on default(employee778701d1ea8c43aeb28e5ede95ebecb1job_title) USING GSI
[2022-06-12 10:57:24,126] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee778701d1ea8c43aeb28e5ede95ebecb1job_title) USING GSI
[2022-06-12 10:57:24,129] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employee778701d1ea8c43aeb28e5ede95ebecb1job_title%29+USING+GSI
[2022-06-12 10:57:24,151] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.436668ms
[2022-06-12 10:57:25,157] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee778701d1ea8c43aeb28e5ede95ebecb1job_title'
[2022-06-12 10:57:25,160] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee778701d1ea8c43aeb28e5ede95ebecb1job_title%27
[2022-06-12 10:57:25,169] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.628146ms
[2022-06-12 10:57:26,174] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee778701d1ea8c43aeb28e5ede95ebecb1job_title'
[2022-06-12 10:57:26,177] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee778701d1ea8c43aeb28e5ede95ebecb1job_title%27
[2022-06-12 10:57:26,185] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.293404ms
[2022-06-12 10:57:26,699] - [basetestcase:2754] INFO - delete 0.0 to default documents...
[2022-06-12 10:57:26,728] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:57:27,231] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:57:28,187] - [task:3140] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:28,195] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:57:28,199] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-06-12 10:57:28,203] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.273372ms
[2022-06-12 10:57:28,203] - [task:3150] INFO - {'requestID': '04605346-1cd3-4fa5-a91f-692d5b43b87a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee778701d1ea8c43aeb28e5ede95ebecb1job_title', 'index_id': '53f62c0cda796435', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.273372ms', 'executionTime': '2.202573ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-06-12 10:57:28,203] - [task:3151] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:28,203] - [task:3181] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:28,204] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:57:28,204] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 10:57:28,204] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-06-12 10:57:28,205] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 10:57:28,205] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 10:57:28,205] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:57:28,205] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:57:29,205] - [task:3140] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:29,209] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:57:29,213] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-06-12 10:57:29,276] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.586806ms
[2022-06-12 10:57:29,276] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 10:57:29,277] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 10:57:30,049] - [task:3151] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:30,049] - [task:3181] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:57:31,055] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee778701d1ea8c43aeb28e5ede95ebecb1job_title'
[2022-06-12 10:57:31,058] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee778701d1ea8c43aeb28e5ede95ebecb1job_title%27
[2022-06-12 10:57:31,068] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.49746ms
[2022-06-12 10:57:31,071] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee778701d1ea8c43aeb28e5ede95ebecb1job_title ON default USING GSI
[2022-06-12 10:57:31,074] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employee778701d1ea8c43aeb28e5ede95ebecb1job_title+ON+default+USING+GSI
[2022-06-12 10:57:31,125] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.617853ms
[2022-06-12 10:57:31,129] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee778701d1ea8c43aeb28e5ede95ebecb1job_title'
[2022-06-12 10:57:31,134] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee778701d1ea8c43aeb28e5ede95ebecb1job_title%27
[2022-06-12 10:57:31,137] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.595477ms
[2022-06-12 10:57:31,137] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fffb5f00-2a7d-454d-81f7-1a1d4cd619bf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.595477ms', 'executionTime': '1.534008ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:57:31,152] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:31,155] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:31,155] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:31,505] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:31,514] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:57:31,514] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 10:57:31,567] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:57:31,576] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:31,579] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:31,579] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:31,880] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:31,889] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:57:31,889] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 10:57:31,941] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:57:31,950] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:31,951] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 10:57:31,951] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:57:31,955] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:57:31,959] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:57:31,965] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.250838ms
[2022-06-12 10:57:31,969] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:57:31,972] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 10:57:32,022] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.87018ms
[2022-06-12 10:57:32,045] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:57:32,045] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.72995402774496, 'mem_free': 13205753856, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:57:32,045] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:57:32,050] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:32,050] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:32,368] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:32,373] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:32,373] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:32,676] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:32,681] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:32,682] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:32,981] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:32,986] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:32,986] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:33,302] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:37,444] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:57:37,473] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 10:57:38,504] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 10:57:38,507] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 10:57:38,507] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 10:57:38,512] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:57:38,517] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:57:38,523] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:57:38,524] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 10:57:38,534] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 10:57:38,535] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 10:57:38,538] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,552] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 10:57:38,552] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,557] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 10:57:38,562] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 10:57:38,563] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,566] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 10:57:38,570] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 10:57:38,571] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,576] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 10:57:38,579] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 10:57:38,580] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,587] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 10:57:38,587] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:57:38,587] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 10:57:38,588] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 69.839s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_3'}
[2022-06-12 10:57:38,653] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:38,653] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:38,920] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:38,928] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:57:38,937] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 10:57:38,938] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-06-12 10:57:38,938] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 10:57:39,656] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 10:57:39,660] - [task:159] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13316100096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:57:39,665] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 10:57:39,665] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 10:57:39,665] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 10:57:39,671] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 10:57:39,681] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 10:57:39,681] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 10:57:39,685] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 10:57:39,686] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 10:57:39,686] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 10:57:39,686] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 10:57:39,689] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:57:39,693] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:39,693] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:40,008] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:40,009] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:57:40,060] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:57:40,061] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:57:40,066] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:57:40,066] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:57:40,068] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:57:40,081] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 10:57:40,084] - [task:159] INFO -  {'uptime': '223', 'memoryTotal': 15466930176, 'memoryFree': 13265948672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:57:40,088] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:57:40,094] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 10:57:40,094] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 10:57:40,097] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:57:40,101] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:40,101] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:40,392] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:40,393] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:57:40,437] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:57:40,438] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:57:40,442] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:57:40,443] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:57:40,445] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:57:40,457] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 10:57:40,460] - [task:159] INFO -  {'uptime': '217', 'memoryTotal': 15466930176, 'memoryFree': 13265915904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:57:40,464] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:57:40,468] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 10:57:40,469] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 10:57:40,471] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:57:40,477] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:40,477] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:40,744] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:40,745] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:57:40,792] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:57:40,793] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:57:40,798] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:57:40,799] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:57:40,801] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:57:40,813] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 10:57:40,816] - [task:159] INFO -  {'uptime': '218', 'memoryTotal': 15466930176, 'memoryFree': 13316239360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:57:40,821] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:57:40,825] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 10:57:40,826] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 10:57:40,828] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:57:40,833] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:40,833] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:41,106] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:41,107] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:57:41,152] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:57:41,153] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:57:41,159] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:57:41,159] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:57:41,162] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:57:41,170] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 10:57:41,280] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:57:46,283] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 10:57:46,295] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 10:57:46,300] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:46,300] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:46,587] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:46,589] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:57:46,632] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:57:46,633] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:57:46,633] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 10:57:47,191] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 10:57:47,221] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 10:57:47,221] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 10:57:47,789] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:57:47,835] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:57:47,879] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 10:57:47,881] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:57:47,890] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:47,890] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:48,191] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:48,196] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:48,196] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:48,460] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:48,464] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:48,465] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:48,734] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:48,740] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:48,740] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:49,131] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:52,824] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:57:52,824] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.96224715689947, 'mem_free': 13323702272, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:57:52,824] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:57:52,825] - [basetestcase:563] INFO - Time to execute basesetup : 14.173784971237183
[2022-06-12 10:57:52,830] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:57:52,830] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:57:52,838] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:57:52,838] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:57:52,845] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:57:52,846] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:57:52,852] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:57:52,852] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:57:52,858] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:57:52,869] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 10:57:52,869] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 10:57:52,870] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:57:57,881] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 10:57:57,885] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:57:57,885] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:57:58,163] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:57:59,115] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 10:57:59,143] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:58:01,228] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:58:01,244] - [newtuq:82] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-06-12 10:58:02,024] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:02,024] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 10:58:02,024] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 10:58:32,055] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 10:58:32,059] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:58:32,062] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:58:32,083] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.465662ms
[2022-06-12 10:58:32,083] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4e5fbe5c-d4fc-4560-bd84-a0d48aaa5091', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.465662ms', 'executionTime': '18.380307ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:58:32,083] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 10:58:32,087] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 10:58:32,091] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 10:58:32,534] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 441.399821ms
[2022-06-12 10:58:32,535] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 10:58:32,540] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:58:32,547] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:58:32,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.510679ms
[2022-06-12 10:58:32,569] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:32,569] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:32,998] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:33,018] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:33,022] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 10:58:33,033] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 10:58:33,034] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 10:58:33,042] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 10:58:33,052] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:34,018] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeedfe22db223d14a0e86a7c66c96f708cbjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 10:58:34,021] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeedfe22db223d14a0e86a7c66c96f708cbjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 10:58:34,085] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.449092ms
[2022-06-12 10:58:34,086] - [base_gsi:282] INFO - BUILD INDEX on default(employeedfe22db223d14a0e86a7c66c96f708cbjob_title) USING GSI
[2022-06-12 10:58:35,090] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeedfe22db223d14a0e86a7c66c96f708cbjob_title) USING GSI
[2022-06-12 10:58:35,093] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employeedfe22db223d14a0e86a7c66c96f708cbjob_title%29+USING+GSI
[2022-06-12 10:58:35,120] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.174587ms
[2022-06-12 10:58:36,125] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedfe22db223d14a0e86a7c66c96f708cbjob_title'
[2022-06-12 10:58:36,129] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedfe22db223d14a0e86a7c66c96f708cbjob_title%27
[2022-06-12 10:58:36,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.075927ms
[2022-06-12 10:58:37,143] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedfe22db223d14a0e86a7c66c96f708cbjob_title'
[2022-06-12 10:58:37,147] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedfe22db223d14a0e86a7c66c96f708cbjob_title%27
[2022-06-12 10:58:37,159] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.266057ms
[2022-06-12 10:58:37,691] - [basetestcase:2754] INFO - update 0.0 to default documents...
[2022-06-12 10:58:37,719] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:58:38,951] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:58:39,161] - [task:3140] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:39,168] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:58:39,171] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-06-12 10:58:39,175] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.562749ms
[2022-06-12 10:58:39,176] - [task:3150] INFO - {'requestID': 'bea8a18c-f2fc-47b2-b3fc-dbc4dd30ce40', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeedfe22db223d14a0e86a7c66c96f708cbjob_title', 'index_id': '549d4d46285823be', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.562749ms', 'executionTime': '2.470195ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-06-12 10:58:39,176] - [task:3151] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:39,176] - [task:3181] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:39,177] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:58:39,177] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 10:58:39,177] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-06-12 10:58:39,178] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 10:58:39,178] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 10:58:39,178] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:58:39,178] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:58:40,178] - [task:3140] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:40,183] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:58:40,187] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-06-12 10:58:40,262] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.354221ms
[2022-06-12 10:58:40,262] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 10:58:40,263] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 10:58:41,027] - [task:3151] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:41,028] - [task:3181] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:58:42,033] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedfe22db223d14a0e86a7c66c96f708cbjob_title'
[2022-06-12 10:58:42,037] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedfe22db223d14a0e86a7c66c96f708cbjob_title%27
[2022-06-12 10:58:42,045] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.75069ms
[2022-06-12 10:58:42,049] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeedfe22db223d14a0e86a7c66c96f708cbjob_title ON default USING GSI
[2022-06-12 10:58:42,052] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeedfe22db223d14a0e86a7c66c96f708cbjob_title+ON+default+USING+GSI
[2022-06-12 10:58:42,110] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.160223ms
[2022-06-12 10:58:42,115] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedfe22db223d14a0e86a7c66c96f708cbjob_title'
[2022-06-12 10:58:42,119] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedfe22db223d14a0e86a7c66c96f708cbjob_title%27
[2022-06-12 10:58:42,122] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.68645ms
[2022-06-12 10:58:42,123] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3b941361-f09a-43e5-900d-c5d2c145298e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.68645ms', 'executionTime': '1.626041ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:58:42,144] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:42,148] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:42,149] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:42,505] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:42,514] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:58:42,514] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 10:58:42,577] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:58:42,586] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:42,589] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:42,590] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:42,943] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:42,952] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:58:42,953] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 10:58:43,020] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:58:43,030] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:58:43,030] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 10:58:43,030] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:58:43,034] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:58:43,038] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:58:43,047] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.736939ms
[2022-06-12 10:58:43,051] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:58:43,054] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 10:58:43,112] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.878976ms
[2022-06-12 10:58:43,129] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:58:43,130] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.87934571473232, 'mem_free': 13206507520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:58:43,130] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:58:43,135] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:43,136] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:43,493] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:43,498] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:43,498] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:43,849] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:43,854] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:43,854] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:44,405] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:44,412] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:44,413] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:45,054] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:49,298] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:58:49,324] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 10:58:49,627] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 10:58:49,629] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 10:58:49,630] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 10:58:49,635] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:58:49,641] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:58:49,646] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 10:58:49,647] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 10:58:49,657] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 10:58:49,658] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 10:58:49,662] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:49,675] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 10:58:49,675] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:49,678] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 10:58:49,682] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 10:58:49,682] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:49,685] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 10:58:49,689] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 10:58:49,689] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:49,692] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 10:58:49,695] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 10:58:49,695] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:49,698] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 10:58:49,698] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:58:49,698] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 10:58:49,699] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 71.054s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_4'}
[2022-06-12 10:58:49,756] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:49,756] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:50,096] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:50,104] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 10:58:50,113] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 10:58:50,113] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-06-12 10:58:50,114] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 10:58:50,759] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 10:58:50,763] - [task:159] INFO -  {'uptime': '294', 'memoryTotal': 15466930176, 'memoryFree': 13272145920, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:58:50,768] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 10:58:50,768] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 10:58:50,769] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 10:58:50,778] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 10:58:50,792] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 10:58:50,793] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 10:58:50,796] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 10:58:50,797] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 10:58:50,798] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 10:58:50,798] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 10:58:50,803] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:58:50,807] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:50,807] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:51,141] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:51,142] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:58:51,197] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:58:51,198] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:58:51,202] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:58:51,203] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:58:51,205] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:58:51,217] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 10:58:51,219] - [task:159] INFO -  {'uptime': '293', 'memoryTotal': 15466930176, 'memoryFree': 13196222464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:58:51,223] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:58:51,228] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 10:58:51,229] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 10:58:51,232] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:58:51,236] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:51,236] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:51,565] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:51,566] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:58:51,620] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:58:51,621] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:58:51,626] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:58:51,627] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:58:51,629] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:58:51,640] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 10:58:51,643] - [task:159] INFO -  {'uptime': '288', 'memoryTotal': 15466930176, 'memoryFree': 13197058048, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:58:51,647] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:58:51,654] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 10:58:51,654] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 10:58:51,656] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:58:51,663] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:51,663] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:51,997] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:51,998] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:58:52,050] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:58:52,051] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:58:52,056] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:58:52,057] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:58:52,059] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:58:52,073] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 10:58:52,076] - [task:159] INFO -  {'uptime': '289', 'memoryTotal': 15466930176, 'memoryFree': 13204082688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 10:58:52,081] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 10:58:52,085] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 10:58:52,085] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 10:58:52,087] - [rest_client:1153] INFO - --> status:True
[2022-06-12 10:58:52,091] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:52,091] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:52,475] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:52,476] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:58:52,545] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:58:52,546] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:58:52,551] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 10:58:52,552] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 10:58:52,554] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 10:58:52,563] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 10:58:52,684] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:58:57,690] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 10:58:57,703] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 10:58:57,706] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:57,706] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:58:58,074] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:58:58,076] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 10:58:58,139] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 10:58:58,140] - [remote_util:5233] INFO - b'ok'
[2022-06-12 10:58:58,140] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 10:58:58,581] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 10:58:58,612] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 10:58:58,612] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 10:58:59,178] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:58:59,291] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:58:59,888] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 10:58:59,891] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-06-12 10:58:59,901] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:58:59,902] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:00,289] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:00,293] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:00,293] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:00,616] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:00,621] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:00,621] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:01,018] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:01,025] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:01,026] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:01,592] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:05,797] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:59:05,798] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.888166254362, 'mem_free': 13277982720, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:59:05,798] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:59:05,798] - [basetestcase:563] INFO - Time to execute basesetup : 16.04481530189514
[2022-06-12 10:59:05,804] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:59:05,804] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:59:05,812] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:59:05,812] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:59:05,820] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:59:05,820] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:59:05,828] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 10:59:05,828] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 10:59:05,833] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:05,847] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 10:59:05,848] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 10:59:05,848] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 10:59:10,860] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 10:59:10,864] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:10,864] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:11,198] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:12,184] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 10:59:12,211] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:59:14,233] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:59:14,251] - [newtuq:82] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-06-12 10:59:14,867] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:14,868] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 10:59:14,868] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 10:59:44,894] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 10:59:44,899] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:59:44,903] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:59:44,924] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.243209ms
[2022-06-12 10:59:44,924] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bde637b5-86d0-4404-888f-0d08deaf9e44', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.243209ms', 'executionTime': '19.167342ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:59:44,925] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 10:59:44,928] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 10:59:44,932] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 10:59:45,327] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 393.444708ms
[2022-06-12 10:59:45,328] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 10:59:45,337] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:59:45,341] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:59:45,351] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.576355ms
[2022-06-12 10:59:45,366] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:45,366] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:45,892] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:45,910] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:45,913] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 10:59:45,924] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 10:59:45,925] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 10:59:45,934] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 10:59:45,943] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:46,909] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea6af789522ce409da3c56a305f047df7job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 10:59:46,913] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeea6af789522ce409da3c56a305f047df7job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 10:59:46,976] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.64784ms
[2022-06-12 10:59:46,977] - [base_gsi:282] INFO - BUILD INDEX on default(employeea6af789522ce409da3c56a305f047df7job_title) USING GSI
[2022-06-12 10:59:47,982] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea6af789522ce409da3c56a305f047df7job_title) USING GSI
[2022-06-12 10:59:47,985] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employeea6af789522ce409da3c56a305f047df7job_title%29+USING+GSI
[2022-06-12 10:59:48,009] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.529497ms
[2022-06-12 10:59:49,014] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6af789522ce409da3c56a305f047df7job_title'
[2022-06-12 10:59:49,018] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6af789522ce409da3c56a305f047df7job_title%27
[2022-06-12 10:59:49,027] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.148894ms
[2022-06-12 10:59:50,032] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6af789522ce409da3c56a305f047df7job_title'
[2022-06-12 10:59:50,036] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6af789522ce409da3c56a305f047df7job_title%27
[2022-06-12 10:59:50,045] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.425464ms
[2022-06-12 10:59:50,600] - [basetestcase:2754] INFO - update 0.0 to default documents...
[2022-06-12 10:59:50,628] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:59:51,929] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 10:59:51,974] - [data_helper:305] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-06-12 10:59:51,974] - [data_helper:306] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 10:59:52,000] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-06-12 10:59:52,001] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-06-12 10:59:52,001] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2604433963, 0, b'')
[2022-06-12 10:59:52,047] - [task:3140] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:52,050] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:59:52,054] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-06-12 10:59:52,059] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.163125ms
[2022-06-12 10:59:52,059] - [task:3150] INFO - {'requestID': '7fee6d68-bc67-487d-8bf9-6a02cbc438f7', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea6af789522ce409da3c56a305f047df7job_title', 'index_id': '2bd20c7c0babdc4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.163125ms', 'executionTime': '3.078119ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-06-12 10:59:52,059] - [task:3151] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:52,060] - [task:3181] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:52,060] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:59:52,060] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 10:59:52,060] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-06-12 10:59:52,061] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 10:59:52,061] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 10:59:52,061] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:59:52,061] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 10:59:53,061] - [task:3140] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:53,066] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 10:59:53,069] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-06-12 10:59:53,138] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.852726ms
[2022-06-12 10:59:53,138] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 10:59:53,139] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 10:59:53,894] - [task:3151] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:53,895] - [task:3181] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 10:59:54,900] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6af789522ce409da3c56a305f047df7job_title'
[2022-06-12 10:59:54,903] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6af789522ce409da3c56a305f047df7job_title%27
[2022-06-12 10:59:54,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.576062ms
[2022-06-12 10:59:54,915] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea6af789522ce409da3c56a305f047df7job_title ON default USING GSI
[2022-06-12 10:59:54,918] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeea6af789522ce409da3c56a305f047df7job_title+ON+default+USING+GSI
[2022-06-12 10:59:54,956] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.385206ms
[2022-06-12 10:59:54,961] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea6af789522ce409da3c56a305f047df7job_title'
[2022-06-12 10:59:54,965] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea6af789522ce409da3c56a305f047df7job_title%27
[2022-06-12 10:59:54,970] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.500683ms
[2022-06-12 10:59:54,970] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '45038a22-95f1-430a-a5d0-7aa571705a13', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '2.500683ms', 'executionTime': '2.419451ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 10:59:54,984] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:54,988] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:54,988] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:55,401] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:55,411] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:59:55,411] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 10:59:55,480] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:59:55,489] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:55,493] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:55,493] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:55,892] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:55,902] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 10:59:55,902] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 10:59:55,972] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 10:59:55,983] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 10:59:55,983] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 10:59:55,983] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:59:55,987] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 10:59:55,991] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 10:59:56,001] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.436966ms
[2022-06-12 10:59:56,005] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 10:59:56,009] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 10:59:56,082] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.867362ms
[2022-06-12 10:59:56,103] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 10:59:56,103] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.38343984976314, 'mem_free': 13147226112, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 10:59:56,104] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 10:59:56,109] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:56,110] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:56,519] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:56,524] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:56,524] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:56,918] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:56,925] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:56,926] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:57,629] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 10:59:57,637] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 10:59:57,638] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 10:59:58,343] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:03,145] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:00:03,173] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:00:04,037] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:00:04,040] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:00:04,040] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:00:04,047] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:00:04,052] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:00:04,058] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:00:04,059] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:00:04,069] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:00:04,070] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:00:04,073] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,085] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:00:04,085] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,089] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:00:04,092] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:00:04,092] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,095] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:00:04,099] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:00:04,099] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,102] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:00:04,105] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:00:04,105] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,108] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:00:04,108] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:00:04,108] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:00:04,109] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 74.361s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_5'}
[2022-06-12 11:00:04,169] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:04,169] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:04,583] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:04,591] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:00:04,602] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:00:04,602] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-06-12 11:00:04,603] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:00:05,172] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:00:05,176] - [task:159] INFO -  {'uptime': '369', 'memoryTotal': 15466930176, 'memoryFree': 13228163072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:00:05,181] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:00:05,181] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:00:05,181] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:00:05,193] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:00:05,201] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:00:05,202] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:00:05,204] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:00:05,206] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:00:05,206] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:00:05,206] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:00:05,208] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:00:05,213] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:05,213] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:05,630] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:05,631] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:00:05,697] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:00:05,698] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:00:05,703] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:00:05,703] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:00:05,706] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:00:05,719] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:00:05,722] - [task:159] INFO -  {'uptime': '368', 'memoryTotal': 15466930176, 'memoryFree': 13228470272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:00:05,726] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:00:05,731] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:00:05,731] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:00:05,734] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:00:05,739] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:05,739] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:06,148] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:06,149] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:00:06,214] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:00:06,215] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:00:06,222] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:00:06,223] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:00:06,225] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:00:06,236] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:00:06,240] - [task:159] INFO -  {'uptime': '363', 'memoryTotal': 15466930176, 'memoryFree': 13130330112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:00:06,244] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:00:06,248] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:00:06,248] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:00:06,251] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:00:06,255] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:06,255] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:06,657] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:06,658] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:00:06,722] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:00:06,723] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:00:06,729] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:00:06,730] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:00:06,732] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:00:06,743] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:00:06,746] - [task:159] INFO -  {'uptime': '364', 'memoryTotal': 15466930176, 'memoryFree': 13228777472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:00:06,751] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:00:06,755] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:00:06,755] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:00:06,758] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:00:06,762] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:06,762] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:07,170] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:07,171] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:00:07,232] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:00:07,233] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:00:07,237] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:00:07,238] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:00:07,240] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:00:07,249] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:00:07,358] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:00:12,362] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:00:12,378] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:00:12,383] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:12,384] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:12,807] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:12,809] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:00:12,877] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:00:12,878] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:00:12,878] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:00:13,273] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:00:13,306] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:00:13,306] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:00:13,972] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:00:14,015] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:00:14,598] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:00:14,604] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:00:14,614] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:14,615] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:15,060] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:15,065] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:15,065] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:15,462] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:15,467] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:15,467] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:16,105] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:16,117] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:16,117] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:16,818] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:21,327] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:00:21,327] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.82249514665903, 'mem_free': 13231071232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:00:21,328] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:00:21,328] - [basetestcase:563] INFO - Time to execute basesetup : 17.161388635635376
[2022-06-12 11:00:21,333] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:00:21,333] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:00:21,341] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:00:21,341] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:00:21,348] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:00:21,349] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:00:21,355] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:00:21,356] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:00:21,360] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:00:21,378] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:00:21,378] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:00:21,379] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:00:26,388] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:00:26,394] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:00:26,394] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:00:26,774] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:00:27,771] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:00:27,798] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:00:29,742] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:00:29,765] - [newtuq:82] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-06-12 11:00:30,759] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:00:30,759] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:00:30,759] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:01:00,788] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:01:00,792] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:01:00,905] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:01:00,927] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.046826ms
[2022-06-12 11:01:00,927] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '98f7175b-8866-4f71-9d22-ed2461ae25e4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.046826ms', 'executionTime': '18.96734ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:01:00,927] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:01:00,931] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:01:00,934] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:01:01,352] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 415.196348ms
[2022-06-12 11:01:01,352] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:01:01,356] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:01:01,363] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:01:01,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.926666ms
[2022-06-12 11:01:01,392] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:01,393] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:01,930] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:01,950] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:01,953] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:01:01,967] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:01:01,967] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:01:01,978] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:01:01,991] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:02,950] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee22ab10ca42df46329ede8ec955f9b88cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:01:02,953] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employee22ab10ca42df46329ede8ec955f9b88cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:01:03,018] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.828229ms
[2022-06-12 11:01:03,019] - [base_gsi:282] INFO - BUILD INDEX on default(employee22ab10ca42df46329ede8ec955f9b88cjob_title) USING GSI
[2022-06-12 11:01:04,024] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee22ab10ca42df46329ede8ec955f9b88cjob_title) USING GSI
[2022-06-12 11:01:04,028] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employee22ab10ca42df46329ede8ec955f9b88cjob_title%29+USING+GSI
[2022-06-12 11:01:04,051] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.863787ms
[2022-06-12 11:01:05,056] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee22ab10ca42df46329ede8ec955f9b88cjob_title'
[2022-06-12 11:01:05,060] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee22ab10ca42df46329ede8ec955f9b88cjob_title%27
[2022-06-12 11:01:05,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.354932ms
[2022-06-12 11:01:06,075] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee22ab10ca42df46329ede8ec955f9b88cjob_title'
[2022-06-12 11:01:06,079] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee22ab10ca42df46329ede8ec955f9b88cjob_title%27
[2022-06-12 11:01:06,088] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.833082ms
[2022-06-12 11:01:06,508] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:01:06,536] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:01:09,583] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:01:10,093] - [task:3140] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:10,099] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:01:10,102] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-06-12 11:01:10,107] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.223529ms
[2022-06-12 11:01:10,107] - [task:3150] INFO - {'requestID': '15c3ddb9-64da-4360-af1b-8a3dcca6e0c1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee22ab10ca42df46329ede8ec955f9b88cjob_title', 'index_id': 'f7c70468931167eb', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.223529ms', 'executionTime': '3.139184ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-06-12 11:01:10,108] - [task:3151] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:10,108] - [task:3181] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:10,109] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:01:10,109] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:01:10,109] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-06-12 11:01:10,109] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:01:10,110] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:01:10,110] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 11:01:10,110] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 11:01:11,110] - [task:3140] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:11,115] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:01:11,119] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-06-12 11:01:11,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 94.571427ms
[2022-06-12 11:01:11,231] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:01:11,231] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:01:12,772] - [task:3151] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:12,772] - [task:3181] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:01:13,778] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee22ab10ca42df46329ede8ec955f9b88cjob_title'
[2022-06-12 11:01:13,782] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee22ab10ca42df46329ede8ec955f9b88cjob_title%27
[2022-06-12 11:01:13,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.18632ms
[2022-06-12 11:01:13,795] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee22ab10ca42df46329ede8ec955f9b88cjob_title ON default USING GSI
[2022-06-12 11:01:13,798] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employee22ab10ca42df46329ede8ec955f9b88cjob_title+ON+default+USING+GSI
[2022-06-12 11:01:13,836] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.356218ms
[2022-06-12 11:01:13,844] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee22ab10ca42df46329ede8ec955f9b88cjob_title'
[2022-06-12 11:01:13,849] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee22ab10ca42df46329ede8ec955f9b88cjob_title%27
[2022-06-12 11:01:13,852] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.639557ms
[2022-06-12 11:01:13,852] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '15347f1f-b33c-4d04-8205-09e113ff65e1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.639557ms', 'executionTime': '1.572432ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:01:13,867] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:13,870] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:13,870] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:14,361] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:14,370] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:01:14,370] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:01:14,454] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:01:14,464] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:14,467] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:14,467] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:14,934] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:14,943] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:01:14,944] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:01:15,033] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:01:15,046] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:15,046] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:01:15,046] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:01:15,050] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:01:15,054] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:01:15,063] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.784099ms
[2022-06-12 11:01:15,067] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:01:15,072] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:01:15,127] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.134926ms
[2022-06-12 11:01:15,143] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:01:15,143] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.3434762896781, 'mem_free': 13006192640, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:01:15,143] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:01:15,149] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:15,149] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:15,628] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:15,633] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:15,633] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:16,109] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:16,115] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:16,115] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:16,684] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:16,697] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:16,697] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:17,557] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:23,360] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:01:23,388] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:01:23,740] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:01:23,743] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:01:23,743] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:01:23,748] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:01:23,754] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:01:23,759] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:01:23,760] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:01:23,770] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:01:23,771] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:01:23,774] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:23,787] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:01:23,787] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:23,791] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:01:23,795] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:01:23,795] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:23,798] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:01:23,801] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:01:23,802] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:23,804] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:01:23,807] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:01:23,808] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:23,811] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:01:23,811] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:01:23,812] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:01:23,812] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 79.653s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_6'}
[2022-06-12 11:01:23,877] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:23,877] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:24,347] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:24,354] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:01:24,365] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:01:24,365] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-06-12 11:01:24,365] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:01:24,880] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:01:24,883] - [task:159] INFO -  {'uptime': '448', 'memoryTotal': 15466930176, 'memoryFree': 13169786880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:01:24,888] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:01:24,888] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:01:24,888] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:01:24,899] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:01:24,916] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:01:24,916] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:01:24,923] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:01:24,924] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:01:24,924] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:01:24,925] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:01:24,927] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:01:24,932] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:24,932] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:25,407] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:25,408] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:01:25,482] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:01:25,483] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:01:25,489] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:01:25,489] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:01:25,492] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:01:25,504] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:01:25,507] - [task:159] INFO -  {'uptime': '444', 'memoryTotal': 15466930176, 'memoryFree': 13060935680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:01:25,511] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:01:25,516] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:01:25,516] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:01:25,519] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:01:25,523] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:25,523] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:26,001] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:26,002] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:01:26,073] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:01:26,074] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:01:26,079] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:01:26,080] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:01:26,082] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:01:26,092] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:01:26,095] - [task:159] INFO -  {'uptime': '443', 'memoryTotal': 15466930176, 'memoryFree': 13061312512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:01:26,099] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:01:26,104] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:01:26,105] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:01:26,107] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:01:26,113] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:26,113] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:26,570] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:26,572] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:01:26,642] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:01:26,643] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:01:26,648] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:01:26,649] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:01:26,651] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:01:26,663] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:01:26,665] - [task:159] INFO -  {'uptime': '444', 'memoryTotal': 15466930176, 'memoryFree': 13175504896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:01:26,670] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:01:26,674] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:01:26,674] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:01:26,677] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:01:26,682] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:26,682] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:27,176] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:27,177] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:01:27,247] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:01:27,248] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:01:27,254] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:01:27,254] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:01:27,256] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:01:27,266] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:01:27,387] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:01:32,390] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:01:32,404] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:01:32,409] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:32,409] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:32,900] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:32,901] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:01:32,979] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:01:32,980] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:01:32,980] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:01:33,286] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:01:33,318] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:01:33,318] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:01:33,888] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:01:33,935] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:01:33,983] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:01:33,985] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:01:33,994] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:33,994] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:34,517] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:34,522] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:34,522] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:34,981] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:35,044] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:35,045] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:35,847] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:35,855] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:35,855] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:36,674] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:41,773] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:01:41,773] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.60618934338772, 'mem_free': 13175537664, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:01:41,774] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:01:41,774] - [basetestcase:563] INFO - Time to execute basesetup : 17.89939022064209
[2022-06-12 11:01:41,779] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:01:41,779] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:01:41,787] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:01:41,787] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:01:41,795] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:01:41,795] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:01:41,802] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:01:41,802] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:01:41,807] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:41,824] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:01:41,824] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:01:41,825] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:01:46,833] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:01:46,837] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:01:46,837] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:01:47,299] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:01:48,357] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:01:48,386] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:01:50,211] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:01:50,228] - [newtuq:82] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-06-12 11:01:51,700] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:01:51,700] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:01:51,700] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:02:21,715] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:02:21,720] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:02:21,723] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:02:21,745] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.698133ms
[2022-06-12 11:02:21,745] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2b35fee6-c2a1-472b-9d6a-aabb2e24fd8c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '19.698133ms', 'executionTime': '19.615411ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:02:21,745] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:02:21,749] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:02:21,753] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:02:22,132] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 377.449101ms
[2022-06-12 11:02:22,133] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:02:22,137] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:02:22,148] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:02:22,169] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.257786ms
[2022-06-12 11:02:22,184] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:22,184] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:22,829] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:22,850] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:02:22,854] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:02:22,868] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:02:22,869] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:02:22,877] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:02:22,887] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:02:23,849] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee7863c3986a0c40e5941d01e5f722ca70job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:02:23,853] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employee7863c3986a0c40e5941d01e5f722ca70job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:02:23,910] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.021812ms
[2022-06-12 11:02:23,910] - [base_gsi:282] INFO - BUILD INDEX on default(employee7863c3986a0c40e5941d01e5f722ca70job_title) USING GSI
[2022-06-12 11:02:24,915] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee7863c3986a0c40e5941d01e5f722ca70job_title) USING GSI
[2022-06-12 11:02:24,919] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employee7863c3986a0c40e5941d01e5f722ca70job_title%29+USING+GSI
[2022-06-12 11:02:24,941] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.076049ms
[2022-06-12 11:02:25,946] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7863c3986a0c40e5941d01e5f722ca70job_title'
[2022-06-12 11:02:25,950] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7863c3986a0c40e5941d01e5f722ca70job_title%27
[2022-06-12 11:02:25,959] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.51081ms
[2022-06-12 11:02:26,963] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7863c3986a0c40e5941d01e5f722ca70job_title'
[2022-06-12 11:02:26,967] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7863c3986a0c40e5941d01e5f722ca70job_title%27
[2022-06-12 11:02:26,977] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.989913ms
[2022-06-12 11:02:27,381] - [basetestcase:2754] INFO - update 0.0 to default documents...
[2022-06-12 11:02:27,411] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:02:27,884] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:02:28,520] - [basetestcase:2754] INFO - delete 0.0 to default documents...
[2022-06-12 11:02:28,551] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:02:29,935] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:02:30,582] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:02:30,612] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:02:34,118] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:02:34,359] - [task:3140] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:34,367] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:02:34,371] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-06-12 11:02:34,377] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.428818ms
[2022-06-12 11:02:34,380] - [task:3150] INFO - {'requestID': '74235a8e-0b90-40a2-bb0b-b6dfc8599bbb', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee7863c3986a0c40e5941d01e5f722ca70job_title', 'index_id': '67c918f6c067b315', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.428818ms', 'executionTime': '3.338046ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-06-12 11:02:34,380] - [task:3151] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:34,381] - [task:3181] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:34,382] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:02:34,382] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:02:34,382] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-06-12 11:02:34,382] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:02:34,383] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:02:34,383] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 11:02:34,383] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-06-12 11:02:35,383] - [task:3140] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:35,387] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-06-12 11:02:35,390] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-06-12 11:02:35,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 101.880451ms
[2022-06-12 11:02:35,507] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:02:35,508] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:02:37,033] - [task:3151] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:37,033] - [task:3181] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-06-12 11:02:38,039] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7863c3986a0c40e5941d01e5f722ca70job_title'
[2022-06-12 11:02:38,042] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7863c3986a0c40e5941d01e5f722ca70job_title%27
[2022-06-12 11:02:38,052] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.520816ms
[2022-06-12 11:02:38,056] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee7863c3986a0c40e5941d01e5f722ca70job_title ON default USING GSI
[2022-06-12 11:02:38,059] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employee7863c3986a0c40e5941d01e5f722ca70job_title+ON+default+USING+GSI
[2022-06-12 11:02:38,102] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.536072ms
[2022-06-12 11:02:38,107] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7863c3986a0c40e5941d01e5f722ca70job_title'
[2022-06-12 11:02:38,111] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7863c3986a0c40e5941d01e5f722ca70job_title%27
[2022-06-12 11:02:38,114] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.632906ms
[2022-06-12 11:02:38,114] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e78e3b6f-52ff-4106-ba1e-6334d84cfe29', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.632906ms', 'executionTime': '1.570959ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:02:38,128] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:02:38,133] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:38,134] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:38,743] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:38,753] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:02:38,753] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:02:38,849] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:02:38,858] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:02:38,862] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:38,862] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:39,455] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:39,464] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:02:39,465] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:02:39,561] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:02:39,573] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:02:39,573] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:02:39,573] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:02:39,577] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:02:39,581] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:02:39,590] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.734037ms
[2022-06-12 11:02:39,593] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:02:39,596] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:02:39,649] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.616967ms
[2022-06-12 11:02:39,665] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:02:39,665] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.81373943875037, 'mem_free': 12982517760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:02:39,665] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:02:39,673] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:39,673] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:40,354] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:40,362] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:40,363] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:40,970] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:40,975] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:40,975] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:42,002] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:42,014] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:42,014] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:43,066] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:49,897] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:02:49,924] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:02:50,707] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:02:50,710] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:02:50,710] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:02:50,716] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:02:50,721] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:02:50,726] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:02:50,727] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:02:50,737] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:02:50,738] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:02:50,741] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:50,755] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:02:50,755] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:50,758] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:02:50,762] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:02:50,762] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:50,765] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:02:50,768] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:02:50,769] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:50,772] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:02:50,776] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:02:50,776] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:50,779] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:02:50,779] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-06-12 11:02:50,779] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:02:50,780] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 86.912s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_7'}
[2022-06-12 11:02:50,877] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:50,877] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:51,480] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:51,488] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:02:51,498] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:02:51,498] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-06-12 11:02:51,499] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:02:51,880] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:02:51,885] - [task:159] INFO -  {'uptime': '535', 'memoryTotal': 15466930176, 'memoryFree': 13115727872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:02:51,890] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:02:51,890] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:02:51,890] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:02:51,896] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:02:51,908] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:02:51,908] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:02:51,911] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:02:51,912] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:02:51,912] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:02:51,912] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:02:51,915] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:02:51,919] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:51,919] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:52,528] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:52,529] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:02:52,615] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:02:52,616] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:02:52,621] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:02:52,622] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:02:52,624] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:02:52,637] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:02:52,640] - [task:159] INFO -  {'uptime': '534', 'memoryTotal': 15466930176, 'memoryFree': 13113921536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:02:52,645] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:02:52,650] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:02:52,650] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:02:52,653] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:02:52,658] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:52,658] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:53,214] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:53,216] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:02:53,299] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:02:53,300] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:02:53,305] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:02:53,305] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:02:53,308] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:02:53,319] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:02:53,322] - [task:159] INFO -  {'uptime': '534', 'memoryTotal': 15466930176, 'memoryFree': 13115428864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:02:53,327] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:02:53,332] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:02:53,332] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:02:53,335] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:02:53,341] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:53,341] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:53,889] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:53,890] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:02:53,979] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:02:53,980] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:02:53,985] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:02:53,986] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:02:53,988] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:02:53,999] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:02:54,003] - [task:159] INFO -  {'uptime': '530', 'memoryTotal': 15466930176, 'memoryFree': 13114544128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:02:54,007] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:02:54,011] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:02:54,012] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:02:54,014] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:02:54,019] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:54,019] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:02:54,561] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:02:54,562] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:02:54,647] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:02:54,648] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:02:54,653] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:02:54,653] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:02:54,656] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:02:54,666] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:02:54,802] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:02:59,804] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:02:59,818] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:02:59,823] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:02:59,823] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:00,405] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:00,406] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:03:00,500] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:03:00,501] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:03:00,501] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:03:00,686] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:03:00,718] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:03:00,718] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:03:01,297] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:03:01,344] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:03:01,388] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:03:01,391] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-06-12 11:03:01,400] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:01,400] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:02,004] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:02,009] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:02,010] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:02,795] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:02,802] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:02,802] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:03,766] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:03,778] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:03,778] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:04,753] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:10,665] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:03:10,665] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.77751334919905, 'mem_free': 13130960896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:03:10,665] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:03:10,666] - [basetestcase:563] INFO - Time to execute basesetup : 19.791215419769287
[2022-06-12 11:03:10,671] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:03:10,671] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:03:10,679] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:03:10,679] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:03:10,687] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:03:10,687] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:03:10,695] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:03:10,695] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:03:10,700] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:10,719] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:03:10,720] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:03:10,720] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:03:15,731] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:03:15,734] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:15,734] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:16,324] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:17,448] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:03:17,476] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:03:20,085] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:03:20,116] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:20,116] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:03:20,116] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:03:50,123] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:03:50,126] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:03:50,130] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:03:50,153] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.827483ms
[2022-06-12 11:03:50,153] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '56e79557-c79a-43f8-adbf-b015fadb4068', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.827483ms', 'executionTime': '20.750375ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:03:50,153] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:03:50,157] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:03:50,160] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:03:50,575] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 413.35348ms
[2022-06-12 11:03:50,576] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:03:50,583] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:03:50,588] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:03:50,598] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.917544ms
[2022-06-12 11:03:50,616] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:50,617] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:51,322] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:51,341] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:51,344] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:03:51,357] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:03:51,357] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:03:51,366] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:03:51,367] - [base_gsi:326] INFO - []
[2022-06-12 11:03:52,341] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeeda177104d354792944e3b7525909993job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:03:52,345] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeeeda177104d354792944e3b7525909993job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:03:52,402] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.064345ms
[2022-06-12 11:03:52,406] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeeda177104d354792944e3b7525909993join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:03:52,415] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeeeda177104d354792944e3b7525909993join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:03:52,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.067393ms
[2022-06-12 11:03:52,466] - [base_gsi:282] INFO - BUILD INDEX on default(employeeeda177104d354792944e3b7525909993job_title,employeeeda177104d354792944e3b7525909993join_yr) USING GSI
[2022-06-12 11:03:53,471] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeeda177104d354792944e3b7525909993job_title,employeeeda177104d354792944e3b7525909993join_yr) USING GSI
[2022-06-12 11:03:53,474] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employeeeda177104d354792944e3b7525909993job_title%2Cemployeeeda177104d354792944e3b7525909993join_yr%29+USING+GSI
[2022-06-12 11:03:53,507] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.994618ms
[2022-06-12 11:03:54,512] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993job_title'
[2022-06-12 11:03:54,516] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993job_title%27
[2022-06-12 11:03:54,526] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.892551ms
[2022-06-12 11:03:55,531] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993job_title'
[2022-06-12 11:03:55,534] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993job_title%27
[2022-06-12 11:03:55,543] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.747292ms
[2022-06-12 11:03:55,546] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993join_yr'
[2022-06-12 11:03:55,550] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993join_yr%27
[2022-06-12 11:03:55,552] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.116223ms
[2022-06-12 11:03:56,558] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993job_title'
[2022-06-12 11:03:56,561] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993job_title%27
[2022-06-12 11:03:56,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.428766ms
[2022-06-12 11:03:56,573] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeeda177104d354792944e3b7525909993job_title ON default USING GSI
[2022-06-12 11:03:56,577] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeeeda177104d354792944e3b7525909993job_title+ON+default+USING+GSI
[2022-06-12 11:03:56,615] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.868912ms
[2022-06-12 11:03:56,623] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993join_yr'
[2022-06-12 11:03:56,627] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993join_yr%27
[2022-06-12 11:03:56,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.533104ms
[2022-06-12 11:03:56,638] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeeda177104d354792944e3b7525909993join_yr ON default USING GSI
[2022-06-12 11:03:56,642] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeeeda177104d354792944e3b7525909993join_yr+ON+default+USING+GSI
[2022-06-12 11:03:56,696] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.385499ms
[2022-06-12 11:03:56,701] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993job_title'
[2022-06-12 11:03:56,706] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993job_title%27
[2022-06-12 11:03:56,713] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.63243ms
[2022-06-12 11:03:56,713] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e7f6a46a-4264-4f76-b197-f01974ca0fe9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.63243ms', 'executionTime': '5.573433ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:03:56,718] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeda177104d354792944e3b7525909993join_yr'
[2022-06-12 11:03:56,722] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeda177104d354792944e3b7525909993join_yr%27
[2022-06-12 11:03:56,725] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.217745ms
[2022-06-12 11:03:56,725] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6bb39e1d-20dd-4ae7-b740-5415c2a8f292', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.217745ms', 'executionTime': '1.177426ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:03:56,740] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:56,743] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:56,744] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:57,302] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:57,312] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:03:57,312] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:03:57,397] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:03:57,408] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:57,412] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:57,412] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:57,966] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:57,975] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:03:57,975] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:03:58,066] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:03:58,074] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:03:58,074] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:03:58,074] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:03:58,078] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:03:58,082] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:03:58,092] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.453072ms
[2022-06-12 11:03:58,096] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:03:58,099] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:03:58,154] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.474916ms
[2022-06-12 11:03:58,168] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:03:58,168] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.458324640622303, 'mem_free': 13014302720, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:03:58,169] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:03:58,172] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:58,172] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:58,719] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:58,724] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:58,724] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:03:59,601] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:03:59,614] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:03:59,614] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:00,590] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:00,598] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:00,599] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:01,624] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:07,564] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-06-12 11:04:07,594] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:04:08,166] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:04:08,169] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:04:08,170] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:04:08,174] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:04:08,180] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:04:08,186] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:04:08,187] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:04:08,197] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:04:08,198] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:04:08,202] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,216] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:04:08,216] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,219] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:04:08,224] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:04:08,224] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,227] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:04:08,231] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:04:08,231] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,234] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:04:08,237] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:04:08,237] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,241] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:04:08,241] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-06-12 11:04:08,241] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:04:08,242] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 77.374s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_8'}
[2022-06-12 11:04:08,281] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:08,281] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:08,875] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:08,883] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:04:08,893] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:04:08,894] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-06-12 11:04:08,894] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:04:09,284] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:04:09,288] - [task:159] INFO -  {'uptime': '613', 'memoryTotal': 15466930176, 'memoryFree': 13144580096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:04:09,292] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:04:09,292] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:04:09,293] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:04:09,302] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:04:09,310] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:04:09,310] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:04:09,313] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:04:09,314] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:04:09,314] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:04:09,315] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:04:09,316] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:04:09,321] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:09,321] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:09,891] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:09,892] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:04:09,982] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:04:09,983] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:04:09,987] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:04:09,988] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:04:09,990] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:04:10,003] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:04:10,007] - [task:159] INFO -  {'uptime': '609', 'memoryTotal': 15466930176, 'memoryFree': 13145313280, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:04:10,011] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:04:10,016] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:04:10,017] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:04:10,018] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:04:10,024] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:10,025] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:10,620] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:10,621] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:04:10,708] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:04:10,709] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:04:10,714] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:04:10,714] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:04:10,717] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:04:10,729] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:04:10,733] - [task:159] INFO -  {'uptime': '609', 'memoryTotal': 15466930176, 'memoryFree': 13144748032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:04:10,738] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:04:10,744] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:04:10,744] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:04:10,746] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:04:10,753] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:10,753] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:11,309] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:11,310] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:04:11,396] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:04:11,397] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:04:11,402] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:04:11,402] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:04:11,405] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:04:11,416] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:04:11,419] - [task:159] INFO -  {'uptime': '610', 'memoryTotal': 15466930176, 'memoryFree': 13145210880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:04:11,424] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:04:11,428] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:04:11,428] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:04:11,431] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:04:11,435] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:11,436] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:11,998] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:11,999] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:04:12,081] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:04:12,082] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:04:12,087] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:04:12,088] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:04:12,090] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:04:12,099] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:04:12,223] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:04:17,228] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:04:17,241] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:04:17,245] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:17,245] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:17,789] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:17,790] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:04:17,880] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:04:17,881] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:04:17,881] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:04:18,121] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:04:18,151] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:04:18,151] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:04:18,726] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:04:18,772] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:04:18,962] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:04:18,965] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-06-12 11:04:18,975] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:18,975] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:19,545] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:19,549] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:19,550] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:20,225] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:20,234] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:20,234] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:21,296] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:21,304] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:21,304] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:22,290] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:28,265] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:04:28,266] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.68278178761632, 'mem_free': 13144301568, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:04:28,266] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:04:28,266] - [basetestcase:563] INFO - Time to execute basesetup : 19.987645864486694
[2022-06-12 11:04:28,271] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:04:28,271] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:04:28,279] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:04:28,279] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:04:28,288] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:04:28,288] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:04:28,296] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:04:28,296] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:04:28,301] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:04:28,315] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:04:28,315] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:04:28,315] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:04:33,325] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:04:33,330] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:04:33,330] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:04:33,897] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:04:34,935] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:04:34,966] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:04:37,709] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:04:37,739] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:04:37,739] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:04:37,740] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:05:07,767] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:05:07,771] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:05:07,775] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:05:07,795] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.693043ms
[2022-06-12 11:05:07,795] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd7865285-b20a-4211-b48a-249e54552b46', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.693043ms', 'executionTime': '18.621152ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:05:07,796] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:05:07,799] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:05:07,802] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:05:08,192] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 388.652803ms
[2022-06-12 11:05:08,193] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:05:08,198] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:05:08,206] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:05:08,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.752293ms
[2022-06-12 11:05:08,246] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:08,246] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:08,944] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:08,965] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:08,969] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:05:08,981] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:05:08,981] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:05:08,990] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:05:08,990] - [base_gsi:326] INFO - []
[2022-06-12 11:05:09,964] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:05:09,968] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:05:10,025] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.07739ms
[2022-06-12 11:05:10,026] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr) USING GSI
[2022-06-12 11:05:11,031] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr) USING GSI
[2022-06-12 11:05:11,036] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%29+USING+GSI
[2022-06-12 11:05:11,056] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.457075ms
[2022-06-12 11:05:12,062] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr'
[2022-06-12 11:05:12,066] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%27
[2022-06-12 11:05:12,076] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.454808ms
[2022-06-12 11:05:13,081] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr'
[2022-06-12 11:05:13,084] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%27
[2022-06-12 11:05:13,092] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.937644ms
[2022-06-12 11:05:14,097] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr'
[2022-06-12 11:05:14,101] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%27
[2022-06-12 11:05:14,110] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.673836ms
[2022-06-12 11:05:14,114] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr ON default USING GSI
[2022-06-12 11:05:14,117] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr+ON+default+USING+GSI
[2022-06-12 11:05:14,164] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.916694ms
[2022-06-12 11:05:14,168] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr'
[2022-06-12 11:05:14,172] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb53ce2c4d2ac4bc2a4ccd08055f744e6job_title_join_yr%27
[2022-06-12 11:05:14,175] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.587931ms
[2022-06-12 11:05:14,176] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5f4119b8-b4e8-4796-bf64-27cac0f17de4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.587931ms', 'executionTime': '1.517418ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:05:14,192] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:14,194] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:14,195] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:14,785] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:14,794] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:05:14,794] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:05:14,887] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:05:14,896] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:14,900] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:14,900] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:15,488] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:15,498] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:05:15,498] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:05:15,598] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:05:15,610] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:15,611] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:05:15,611] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:05:15,615] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:05:15,618] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:05:15,628] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.073465ms
[2022-06-12 11:05:15,631] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:05:15,635] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:05:15,671] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.602583ms
[2022-06-12 11:05:15,697] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:05:15,697] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.680754863711275, 'mem_free': 13007323136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:05:15,698] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:05:15,704] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:15,704] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:16,298] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:16,306] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:16,306] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:17,296] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:17,304] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:17,304] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:18,344] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:18,355] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:18,355] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:19,395] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:25,383] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-06-12 11:05:25,414] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:05:26,547] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:05:26,550] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:05:26,551] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:05:26,557] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:05:26,562] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:05:26,567] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:05:26,568] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:05:26,578] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:05:26,579] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:05:26,583] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:26,597] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:05:26,597] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:26,608] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:05:26,613] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:05:26,613] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:26,617] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:05:26,620] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:05:26,620] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:26,623] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:05:26,627] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:05:26,627] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:26,630] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:05:26,630] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-06-12 11:05:26,630] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:05:26,631] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 78.359s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_9'}
[2022-06-12 11:05:26,676] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:26,676] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:27,315] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:27,323] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:05:27,333] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:05:27,333] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-06-12 11:05:27,334] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:05:27,679] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:05:27,683] - [task:159] INFO -  {'uptime': '691', 'memoryTotal': 15466930176, 'memoryFree': 13127839744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:05:27,687] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:05:27,688] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:05:27,688] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:05:27,700] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:05:27,705] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:05:27,705] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:05:27,708] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:05:27,709] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:05:27,709] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:05:27,710] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:05:27,712] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:05:27,717] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:27,717] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:28,314] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:28,315] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:05:28,416] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:05:28,417] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:05:28,422] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:05:28,422] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:05:28,425] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:05:28,437] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:05:28,440] - [task:159] INFO -  {'uptime': '690', 'memoryTotal': 15466930176, 'memoryFree': 13128159232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:05:28,444] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:05:28,448] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:05:28,449] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:05:28,451] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:05:28,456] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:28,456] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:29,043] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:29,044] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:05:29,129] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:05:29,130] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:05:29,134] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:05:29,135] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:05:29,137] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:05:29,150] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:05:29,154] - [task:159] INFO -  {'uptime': '689', 'memoryTotal': 15466930176, 'memoryFree': 13128769536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:05:29,158] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:05:29,164] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:05:29,164] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:05:29,166] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:05:29,172] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:29,173] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:29,743] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:29,744] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:05:29,832] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:05:29,834] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:05:29,839] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:05:29,839] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:05:29,842] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:05:29,853] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:05:29,857] - [task:159] INFO -  {'uptime': '685', 'memoryTotal': 15466930176, 'memoryFree': 13128552448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:05:29,861] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:05:29,867] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:05:29,867] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:05:29,869] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:05:29,873] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:29,873] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:30,451] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:30,453] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:05:30,548] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:05:30,549] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:05:30,554] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:05:30,554] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:05:30,557] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:05:30,567] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:05:30,700] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:05:35,705] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:05:35,719] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:05:35,723] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:35,723] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:36,325] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:36,326] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:05:36,412] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:05:36,413] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:05:36,414] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:05:36,587] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:05:36,621] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:05:36,622] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:05:37,194] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:05:37,236] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:05:37,894] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:05:37,897] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-06-12 11:05:37,905] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:37,905] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:38,524] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:38,528] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:38,528] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:39,275] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:39,284] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:39,284] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:40,294] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:40,302] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:40,302] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:41,317] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:47,377] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:05:47,378] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.23995278346325, 'mem_free': 13128507392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:05:47,378] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:05:47,378] - [basetestcase:563] INFO - Time to execute basesetup : 20.705093383789062
[2022-06-12 11:05:47,384] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:05:47,384] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:05:47,391] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:05:47,392] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:05:47,398] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:05:47,398] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:05:47,406] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:05:47,406] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:05:47,411] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:47,430] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:05:47,430] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:05:47,430] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:05:52,443] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:05:52,446] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:05:52,446] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:05:53,042] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:05:54,207] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:05:54,235] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:05:56,822] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:05:56,854] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:05:56,854] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:05:56,855] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:06:26,883] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:06:26,888] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:06:26,893] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:06:26,920] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.627931ms
[2022-06-12 11:06:26,920] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '58740a80-c408-4b7d-b125-f01d49d13e5a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '24.627931ms', 'executionTime': '24.535143ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:06:26,920] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:06:26,925] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:06:26,932] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:06:27,337] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 402.59076ms
[2022-06-12 11:06:27,337] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:06:27,342] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:06:27,351] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:06:27,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 12.747878ms
[2022-06-12 11:06:27,384] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:27,384] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:28,107] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:28,127] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:28,130] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:06:28,143] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:06:28,143] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:06:28,152] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:06:28,162] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:29,125] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee476f310410ca4fc1bb2bd01479ab1fb9join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:06:29,128] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:06:29,183] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.054469ms
[2022-06-12 11:06:29,183] - [base_gsi:282] INFO - BUILD INDEX on default(`employee476f310410ca4fc1bb2bd01479ab1fb9join_yr`) USING GSI
[2022-06-12 11:06:30,188] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee476f310410ca4fc1bb2bd01479ab1fb9join_yr`) USING GSI
[2022-06-12 11:06:30,192] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%60%29+USING+GSI
[2022-06-12 11:06:30,215] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.806344ms
[2022-06-12 11:06:30,222] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee476f310410ca4fc1bb2bd01479ab1fb9join_yr'
[2022-06-12 11:06:30,228] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%27
[2022-06-12 11:06:30,240] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.28108ms
[2022-06-12 11:06:31,247] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee476f310410ca4fc1bb2bd01479ab1fb9join_yr'
[2022-06-12 11:06:31,251] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%27
[2022-06-12 11:06:31,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.503736ms
[2022-06-12 11:06:32,266] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee476f310410ca4fc1bb2bd01479ab1fb9join_yr'
[2022-06-12 11:06:32,269] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%27
[2022-06-12 11:06:32,278] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.989829ms
[2022-06-12 11:06:33,283] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:06:33,286] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-06-12 11:06:33,291] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.791594ms
[2022-06-12 11:06:33,291] - [base_gsi:504] INFO - {'requestID': '8687d408-f21d-4468-aceb-327c7f6fb57d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee476f310410ca4fc1bb2bd01479ab1fb9join_yr', 'index_id': 'f07edd737c832bac', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.791594ms', 'executionTime': '2.71136ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-06-12 11:06:33,291] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:06:33,292] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:06:33,292] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:06:33,292] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:06:33,293] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:06:33,294] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:06:33,294] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:06:33,295] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:06:33,357] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:06:33,382] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:06:33,386] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-06-12 11:06:33,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 94.869886ms
[2022-06-12 11:06:33,497] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:06:33,498] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:06:35,122] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:35,126] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee476f310410ca4fc1bb2bd01479ab1fb9join_yr'
[2022-06-12 11:06:35,129] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee476f310410ca4fc1bb2bd01479ab1fb9join_yr%27
[2022-06-12 11:06:35,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.509265ms
[2022-06-12 11:06:35,136] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '26da9f93-a757-432a-8802-282eeaf40408', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.509265ms', 'executionTime': '4.43195ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:06:35,145] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:35,147] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:35,148] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:35,789] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:35,800] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:06:35,800] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:06:35,947] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:06:35,963] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:35,971] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:35,971] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:36,645] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:36,653] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:06:36,654] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:06:36,755] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:06:36,767] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:06:36,767] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:06:36,767] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:06:36,771] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:06:36,774] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:06:36,779] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.598546ms
[2022-06-12 11:06:36,779] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6f6e2864-da2e-4c40-8cd0-eba21561c3d8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.598546ms', 'executionTime': '3.513614ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:06:36,785] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:06:36,785] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.828916237861121, 'mem_free': 12972666880, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:06:36,785] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:06:36,789] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:36,789] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:37,432] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:37,437] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:37,438] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:38,363] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:38,372] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:38,372] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:39,465] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:39,478] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:39,478] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:40,566] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:46,828] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-06-12 11:06:46,846] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:06:46,852] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:06:46,856] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:06:46,862] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:06:46,862] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:06:46,870] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:06:46,871] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:06:46,874] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:46,887] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:06:46,887] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:46,891] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:06:46,894] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:06:46,894] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:46,897] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:06:46,900] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:06:46,900] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:46,903] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:06:46,906] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:06:46,906] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:46,909] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:06:46,909] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-06-12 11:06:46,910] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:06:46,910] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 80.244s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_10'}
[2022-06-12 11:06:46,954] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:46,954] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:47,576] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:47,584] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:06:47,593] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:06:47,594] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-06-12 11:06:47,594] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:06:47,956] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:06:47,960] - [task:159] INFO -  {'uptime': '770', 'memoryTotal': 15466930176, 'memoryFree': 12968058880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:06:47,965] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:06:47,965] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:06:47,965] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:06:47,974] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:06:47,986] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:06:47,986] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:06:47,989] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:06:47,989] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:06:47,990] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:06:47,990] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:06:47,992] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:06:47,996] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:47,997] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:48,595] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:48,596] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:06:48,698] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:06:48,699] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:06:48,704] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:06:48,704] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:06:48,706] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:06:48,718] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:06:48,721] - [task:159] INFO -  {'uptime': '770', 'memoryTotal': 15466930176, 'memoryFree': 13112127488, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:06:48,725] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:06:48,730] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:06:48,730] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:06:48,733] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:06:48,740] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:48,740] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:49,336] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:49,337] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:06:49,432] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:06:49,433] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:06:49,438] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:06:49,439] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:06:49,441] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:06:49,452] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:06:49,455] - [task:159] INFO -  {'uptime': '770', 'memoryTotal': 15466930176, 'memoryFree': 13112893440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:06:49,460] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:06:49,464] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:06:49,464] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:06:49,467] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:06:49,471] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:49,471] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:50,065] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:50,067] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:06:50,157] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:06:50,158] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:06:50,164] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:06:50,164] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:06:50,168] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:06:50,183] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:06:50,187] - [task:159] INFO -  {'uptime': '766', 'memoryTotal': 15466930176, 'memoryFree': 13112385536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:06:50,191] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:06:50,195] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:06:50,195] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:06:50,199] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:06:50,204] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:50,204] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:50,821] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:50,823] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:06:50,915] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:06:50,916] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:06:50,921] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:06:50,922] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:06:50,924] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:06:50,932] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:06:51,040] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:06:56,043] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:06:56,058] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:06:56,063] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:56,063] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:56,709] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:56,710] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:06:56,812] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:06:56,814] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:06:56,814] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:06:56,954] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:06:56,986] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:06:56,986] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:06:57,565] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:06:57,608] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:06:57,649] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:06:57,653] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-06-12 11:06:57,663] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:57,663] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:58,306] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:58,311] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:58,311] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:06:59,213] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:06:59,221] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:06:59,221] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:00,306] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:00,319] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:00,320] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:01,419] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:07,936] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:07:07,937] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.99767707810381, 'mem_free': 13110059008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:07:07,937] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:07:07,937] - [basetestcase:563] INFO - Time to execute basesetup : 20.985898971557617
[2022-06-12 11:07:07,943] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:07:07,943] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:07:07,952] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:07:07,952] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:07:07,960] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:07:07,960] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:07:07,968] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:07:07,968] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:07:07,974] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:07,988] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:07:07,989] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:07:07,989] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:07:13,002] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:07:13,006] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:13,006] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:13,618] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:14,790] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:07:14,819] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:07:17,367] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:07:17,398] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:17,399] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:07:17,399] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:07:47,412] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:07:47,417] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:07:47,420] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:07:47,443] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.897843ms
[2022-06-12 11:07:47,443] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '462f68bf-a418-459c-8b91-3efeafd1a06f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '20.897843ms', 'executionTime': '20.818191ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:07:47,443] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:07:47,446] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:07:47,450] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:07:47,893] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 440.28272ms
[2022-06-12 11:07:47,893] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:07:47,899] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:07:47,907] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:07:47,928] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.399494ms
[2022-06-12 11:07:47,950] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:47,950] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:48,693] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:48,712] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:48,715] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:07:48,728] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:07:48,728] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:07:48,738] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:07:48,747] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:49,712] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea40275eebe9b45c5b818066996872506join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:07:49,716] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employeea40275eebe9b45c5b818066996872506join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:07:49,769] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.960923ms
[2022-06-12 11:07:49,770] - [base_gsi:282] INFO - BUILD INDEX on default(`employeea40275eebe9b45c5b818066996872506join_yr`) USING GSI
[2022-06-12 11:07:50,775] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeea40275eebe9b45c5b818066996872506join_yr`) USING GSI
[2022-06-12 11:07:50,778] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeea40275eebe9b45c5b818066996872506join_yr%60%29+USING+GSI
[2022-06-12 11:07:50,803] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.761079ms
[2022-06-12 11:07:50,809] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea40275eebe9b45c5b818066996872506join_yr'
[2022-06-12 11:07:50,814] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea40275eebe9b45c5b818066996872506join_yr%27
[2022-06-12 11:07:50,824] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.545415ms
[2022-06-12 11:07:51,830] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea40275eebe9b45c5b818066996872506join_yr'
[2022-06-12 11:07:51,834] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea40275eebe9b45c5b818066996872506join_yr%27
[2022-06-12 11:07:51,841] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.831715ms
[2022-06-12 11:07:52,847] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea40275eebe9b45c5b818066996872506join_yr'
[2022-06-12 11:07:52,851] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea40275eebe9b45c5b818066996872506join_yr%27
[2022-06-12 11:07:52,860] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.035837ms
[2022-06-12 11:07:53,866] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:07:53,869] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-06-12 11:07:53,874] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.027692ms
[2022-06-12 11:07:53,874] - [base_gsi:504] INFO - {'requestID': '047b8a83-0bda-4952-bd65-907e94abc360', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea40275eebe9b45c5b818066996872506join_yr', 'index_id': '1281d9149b38758a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.027692ms', 'executionTime': '2.944334ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-06-12 11:07:53,875] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:07:53,875] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:53,875] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:07:53,875] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:07:53,876] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:53,876] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:53,876] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:53,876] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:53,943] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:53,968] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:07:53,971] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-06-12 11:07:54,085] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.242179ms
[2022-06-12 11:07:54,085] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:07:54,086] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:07:55,235] - [rest_client:3056] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-06-12 11:07:55,266] - [rest_client:3064] INFO - bucket default updated
[2022-06-12 11:07:55,271] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:07:55,274] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-06-12 11:07:55,277] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.982446ms
[2022-06-12 11:07:55,278] - [base_gsi:504] INFO - {'requestID': 'dd764445-ee76-44a9-885f-9fabc3d69218', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea40275eebe9b45c5b818066996872506join_yr', 'index_id': '1281d9149b38758a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.982446ms', 'executionTime': '1.910832ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-06-12 11:07:55,278] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:07:55,278] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:55,278] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:07:55,279] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:07:55,279] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:55,279] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:55,279] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:55,279] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:07:55,338] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:07:55,353] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:07:55,357] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-06-12 11:07:55,463] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 97.241447ms
[2022-06-12 11:07:55,464] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:07:55,464] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:07:56,605] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea40275eebe9b45c5b818066996872506join_yr ON default USING GSI
[2022-06-12 11:07:56,609] - [rest_client:4170] INFO - query params : statement=DROP+INDEX+employeea40275eebe9b45c5b818066996872506join_yr+ON+default+USING+GSI
[2022-06-12 11:07:56,659] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.691083ms
[2022-06-12 11:07:56,665] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea40275eebe9b45c5b818066996872506join_yr'
[2022-06-12 11:07:56,670] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea40275eebe9b45c5b818066996872506join_yr%27
[2022-06-12 11:07:56,677] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.643627ms
[2022-06-12 11:07:56,677] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7d5fcec0-de01-43cb-a9b2-9febd79d5199', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.643627ms', 'executionTime': '5.590044ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:07:56,691] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:56,693] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:56,694] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:57,347] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:57,356] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:07:57,356] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:07:57,461] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:07:57,473] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:57,476] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:57,476] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:58,108] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:58,117] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:07:58,117] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:07:58,226] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:07:58,234] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:07:58,235] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:07:58,235] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:07:58,238] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:07:58,242] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:07:58,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.093997ms
[2022-06-12 11:07:58,255] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:07:58,259] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:07:58,318] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.074518ms
[2022-06-12 11:07:58,335] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:07:58,335] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 27.00993425926954, 'mem_free': 12916105216, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:07:58,335] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:07:58,338] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:58,339] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:58,987] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:58,992] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:58,992] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:07:59,930] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:07:59,942] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:07:59,942] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:01,065] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:01,077] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:01,077] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:02,241] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:08,879] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-06-12 11:08:08,909] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:08:09,409] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:08:09,415] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:08:09,415] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:08:09,421] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:08:09,428] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:08:09,434] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:08:09,435] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:08:09,446] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:08:09,447] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:08:09,450] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:09,467] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:08:09,467] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:09,470] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:08:09,473] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:08:09,473] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:09,476] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:08:09,479] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:08:09,479] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:09,482] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:08:09,485] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:08:09,485] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:09,488] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:08:09,488] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-06-12 11:08:09,489] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:08:09,490] - [basetestcase:764] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 82.545s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_11'}
[2022-06-12 11:08:09,533] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:09,534] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:10,183] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:10,191] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:08:10,201] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:08:10,201] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-06-12 11:08:10,202] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:08:10,537] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:08:10,541] - [task:159] INFO -  {'uptime': '854', 'memoryTotal': 15466930176, 'memoryFree': 13100896256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:08:10,546] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:08:10,546] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:08:10,546] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:08:10,553] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:08:10,567] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:08:10,568] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:08:10,574] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:08:10,575] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:08:10,576] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:08:10,576] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:08:10,578] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:08:10,582] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:10,583] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:11,253] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:11,254] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:08:11,357] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:08:11,358] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:08:11,363] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:08:11,364] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:08:11,366] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:08:11,379] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:08:11,382] - [task:159] INFO -  {'uptime': '850', 'memoryTotal': 15466930176, 'memoryFree': 13100523520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:08:11,387] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:08:11,392] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:08:11,392] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:08:11,395] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:08:11,400] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:11,400] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:12,075] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:12,077] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:08:12,179] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:08:12,180] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:08:12,185] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:08:12,186] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:08:12,188] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:08:12,200] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:08:12,203] - [task:159] INFO -  {'uptime': '850', 'memoryTotal': 15466930176, 'memoryFree': 13099941888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:08:12,208] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:08:12,213] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:08:12,213] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:08:12,216] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:08:12,222] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:12,222] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:12,886] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:12,888] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:08:12,984] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:08:12,985] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:08:12,990] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:08:12,991] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:08:12,994] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:08:13,011] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:08:13,015] - [task:159] INFO -  {'uptime': '851', 'memoryTotal': 15466930176, 'memoryFree': 13100335104, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:08:13,019] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:08:13,025] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:08:13,026] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:08:13,029] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:08:13,033] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:13,033] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:13,663] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:13,664] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:08:13,764] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:08:13,765] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:08:13,771] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:08:13,772] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:08:13,774] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:08:13,783] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:08:13,909] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:08:18,915] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:08:18,928] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:08:18,934] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:18,934] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:19,603] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:19,604] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:08:19,705] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:08:19,706] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:08:19,707] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:08:19,804] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:08:19,834] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:08:19,835] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:08:20,418] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:08:20,467] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:08:20,699] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:08:20,703] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-06-12 11:08:20,710] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:20,710] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:21,351] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:21,356] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:21,356] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:22,258] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:22,268] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:22,269] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:23,369] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:23,377] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:23,378] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:24,512] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:31,087] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:08:31,088] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.89397519911322, 'mem_free': 13094187008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:08:31,088] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:08:31,088] - [basetestcase:563] INFO - Time to execute basesetup : 21.55726671218872
[2022-06-12 11:08:31,094] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:08:31,094] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:08:31,102] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:08:31,102] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:08:31,110] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:08:31,110] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:08:31,118] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:08:31,118] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:08:31,123] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:08:31,138] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:08:31,138] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:08:31,138] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:08:36,150] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:08:36,153] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:08:36,153] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:08:36,774] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:08:37,836] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:08:37,863] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:08:40,518] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:08:40,553] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:08:40,553] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:08:40,553] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:09:10,579] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:09:10,583] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:09:10,586] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:09:10,609] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.055957ms
[2022-06-12 11:09:10,610] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '85c2c0a7-bfa5-49c9-84b2-823c7cbbb07b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '21.055957ms', 'executionTime': '20.971033ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:09:10,610] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:09:10,614] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:09:10,617] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:09:11,044] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 424.385229ms
[2022-06-12 11:09:11,044] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:09:11,053] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:09:11,059] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:09:11,067] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.7654ms
[2022-06-12 11:09:11,092] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:11,092] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:11,892] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:11,913] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:11,917] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:09:11,929] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:09:11,929] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:09:11,938] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:09:11,948] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:12,912] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee693455b3cbaf49dea013faa5c3a941ecjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-06-12 11:09:12,916] - [rest_client:4170] INFO - query params : statement=CREATE+INDEX+%60employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-06-12 11:09:12,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.397696ms
[2022-06-12 11:09:12,968] - [base_gsi:282] INFO - BUILD INDEX on default(`employee693455b3cbaf49dea013faa5c3a941ecjoin_yr`) USING GSI
[2022-06-12 11:09:13,973] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee693455b3cbaf49dea013faa5c3a941ecjoin_yr`) USING GSI
[2022-06-12 11:09:13,976] - [rest_client:4170] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%60%29+USING+GSI
[2022-06-12 11:09:13,998] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.659111ms
[2022-06-12 11:09:14,011] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr'
[2022-06-12 11:09:14,017] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%27
[2022-06-12 11:09:14,030] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.568373ms
[2022-06-12 11:09:15,036] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr'
[2022-06-12 11:09:15,040] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%27
[2022-06-12 11:09:15,050] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.178648ms
[2022-06-12 11:09:16,055] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr'
[2022-06-12 11:09:16,058] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%27
[2022-06-12 11:09:16,068] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.343246ms
[2022-06-12 11:09:17,074] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:09:17,078] - [rest_client:4170] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-06-12 11:09:17,083] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.893945ms
[2022-06-12 11:09:17,083] - [base_gsi:504] INFO - {'requestID': '90796294-8509-4e1a-81bc-4c4554af7fc0', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr', 'index_id': '548c00c5fd50363a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.893945ms', 'executionTime': '3.816474ms', 'resultCount': 1, 'resultSize': 890, 'serviceLoad': 6}}
[2022-06-12 11:09:17,084] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-06-12 11:09:17,084] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:09:17,084] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-06-12 11:09:17,085] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-06-12 11:09:17,085] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:09:17,085] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:09:17,085] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:09:17,085] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-06-12 11:09:17,167] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-06-12 11:09:17,184] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-06-12 11:09:17,188] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-06-12 11:09:17,301] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 94.930181ms
[2022-06-12 11:09:17,301] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-06-12 11:09:17,302] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-06-12 11:09:19,211] - [basetestcase:833] INFO - sleep for 2 secs.  ...
[2022-06-12 11:09:21,306] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:09:21,340] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:09:21,341] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:09:22,089] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:09:22,146] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:09:22,204] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:09:22,208] - [basetestcase:833] INFO - sleep for 2 secs.  ...
[2022-06-12 11:09:24,214] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:09:24,214] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:24,218] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:09:24,221] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:09:24,221] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:24,224] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:09:24,228] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:09:24,228] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:24,232] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:09:24,235] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:09:24,235] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:24,237] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:09:24,246] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:24,249] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr'
[2022-06-12 11:09:24,253] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%27
[2022-06-12 11:09:24,271] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 16.421322ms
[2022-06-12 11:09:24,271] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4fa2e688-1fb0-4818-942d-0623e47f70c6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '16.421322ms', 'executionTime': '16.349972ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:09:24,275] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee693455b3cbaf49dea013faa5c3a941ecjoin_yr'
[2022-06-12 11:09:24,278] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee693455b3cbaf49dea013faa5c3a941ecjoin_yr%27
[2022-06-12 11:09:24,281] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.177706ms
[2022-06-12 11:09:24,281] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5d391567-00e1-454c-b790-b39cce988245', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.177706ms', 'executionTime': '1.125615ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:09:24,286] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:24,300] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:24,302] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:24,302] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:24,967] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:24,977] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:09:24,977] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:09:25,084] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:09:25,092] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:25,095] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:25,095] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:25,731] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:25,740] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:09:25,741] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:09:25,851] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:09:25,863] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:25,863] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:09:25,863] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:09:25,867] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:09:25,870] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:09:25,880] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.106216ms
[2022-06-12 11:09:25,880] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e8ab1d8f-e1d0-443d-8ace-f804aeed7ce2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.106216ms', 'executionTime': '8.025151ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:09:25,880] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:09:25,883] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:09:25,886] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:09:25,888] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 876.357µs
[2022-06-12 11:09:25,888] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b53edb79-9c5d-4ae8-aca3-2fea1e5d175f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '876.357µs', 'executionTime': '826.822µs', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:09:25,893] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:09:25,893] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.16163210042833, 'mem_free': 12913446912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:09:25,893] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:09:25,900] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:25,901] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:26,573] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:26,578] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:26,579] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:27,350] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:27,358] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:27,358] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:28,499] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:28,511] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:28,512] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:29,678] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:37,034] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-06-12 11:09:37,059] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:09:37,208] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:09:37,211] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:09:37,212] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:09:37,217] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:09:37,223] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:09:37,228] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:09:37,229] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:09:37,240] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:09:37,241] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:09:37,244] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,258] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:09:37,258] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,262] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:09:37,266] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:09:37,266] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,269] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:09:37,272] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:09:37,272] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,276] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:09:37,280] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:09:37,280] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,283] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:09:37,284] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-06-12 11:09:37,284] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:09:37,285] - [basetestcase:764] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 87.760s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Jun-12_10-53-52/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-06-12 11:09:37,298] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:37,298] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:37,980] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:37,988] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:09:37,998] - [rest_client:2639] INFO - Node version in cluster 7.2.0-1350-rel-enterprise
[2022-06-12 11:09:37,998] - [basetestcase:253] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-06-12 11:09:37,998] - [basetestcase:320] INFO - initializing cluster
[2022-06-12 11:09:38,300] - [task:154] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-06-12 11:09:38,304] - [task:159] INFO -  {'uptime': '942', 'memoryTotal': 15466930176, 'memoryFree': 13077065728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:09:38,309] - [task:198] INFO - quota for index service will be 256 MB
[2022-06-12 11:09:38,309] - [task:200] INFO - set index quota to node 127.0.0.1 
[2022-06-12 11:09:38,309] - [rest_client:1262] INFO - pools/default params : indexMemoryQuota=256
[2022-06-12 11:09:38,325] - [rest_client:1250] INFO - pools/default params : memoryQuota=7650
[2022-06-12 11:09:38,332] - [rest_client:1211] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-06-12 11:09:38,332] - [rest_client:1227] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-06-12 11:09:38,335] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-06-12 11:09:38,336] - [rest_client:1233] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-06-12 11:09:38,336] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-06-12 11:09:38,336] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-06-12 11:09:38,338] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:09:38,343] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:38,343] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:39,033] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:39,034] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:09:39,133] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:09:39,134] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:09:39,139] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:09:39,139] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:09:39,144] - [rest_client:1075] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:09:39,156] - [task:154] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-06-12 11:09:39,160] - [task:159] INFO -  {'uptime': '941', 'memoryTotal': 15466930176, 'memoryFree': 13078040576, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:09:39,164] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:09:39,169] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-06-12 11:09:39,169] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-06-12 11:09:39,172] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:09:39,176] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:39,176] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:39,844] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:39,845] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:09:39,962] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:09:39,964] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:09:39,968] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:09:39,969] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:09:39,972] - [rest_client:1075] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:09:39,983] - [task:154] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-06-12 11:09:39,986] - [task:159] INFO -  {'uptime': '940', 'memoryTotal': 15466930176, 'memoryFree': 13077364736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:09:39,991] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:09:39,995] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-06-12 11:09:39,995] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-06-12 11:09:39,998] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:09:40,003] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:40,003] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:40,666] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:40,667] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:09:40,765] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:09:40,766] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:09:40,770] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:09:40,770] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:09:40,772] - [rest_client:1075] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:09:40,783] - [task:154] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-06-12 11:09:40,787] - [task:159] INFO -  {'uptime': '936', 'memoryTotal': 15466930176, 'memoryFree': 13077880832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1350-rel-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-06-12 11:09:40,791] - [rest_client:1250] INFO - pools/default params : memoryQuota=7906
[2022-06-12 11:09:40,799] - [rest_client:1146] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-06-12 11:09:40,799] - [rest_client:1151] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-06-12 11:09:40,801] - [rest_client:1153] INFO - --> status:True
[2022-06-12 11:09:40,805] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:40,805] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:41,465] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:41,466] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:09:41,561] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:09:41,561] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:09:41,566] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-06-12 11:09:41,567] - [rest_client:1285] INFO - settings/indexes params : storageMode=forestdb
[2022-06-12 11:09:41,569] - [rest_client:1075] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-06-12 11:09:41,577] - [basetestcase:2382] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-06-12 11:09:41,683] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:09:46,687] - [basetestcase:2387] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-06-12 11:09:46,699] - [basetestcase:358] INFO - done initializing cluster
[2022-06-12 11:09:46,704] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:46,705] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:47,385] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:47,386] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-06-12 11:09:47,484] - [remote_util:3381] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-06-12 11:09:47,485] - [remote_util:5233] INFO - b'ok'
[2022-06-12 11:09:47,486] - [basetestcase:2975] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-06-12 11:09:47,598] - [rest_client:2994] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-06-12 11:09:47,628] - [rest_client:3019] INFO - 0.03 seconds to create bucket default
[2022-06-12 11:09:47,628] - [bucket_helper:333] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-06-12 11:09:48,194] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:09:48,237] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:09:48,281] - [task:377] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-06-12 11:09:48,284] - [basetestcase:531] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-06-12 11:09:48,292] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:48,293] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:48,991] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:48,996] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:48,996] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:49,921] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:49,928] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:49,929] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:51,084] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:51,092] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:09:51,092] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:09:52,224] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:09:59,063] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:09:59,064] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.79280550802413, 'mem_free': 13072850944, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:09:59,064] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:09:59,064] - [basetestcase:563] INFO - Time to execute basesetup : 21.76945161819458
[2022-06-12 11:09:59,071] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:09:59,071] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:09:59,079] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:09:59,079] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:09:59,087] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:09:59,087] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:09:59,093] - [newtuq:21] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-06-12 11:09:59,093] - [newtuq:26] INFO - current status of 127.0.0.1  is healthy
[2022-06-12 11:09:59,099] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:09:59,119] - [rest_client:2081] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-06-12 11:09:59,119] - [newtuq:36] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-06-12 11:09:59,119] - [basetestcase:833] INFO - sleep for 5 secs.  ...
[2022-06-12 11:10:04,132] - [rest_client:2081] INFO - {'indexer.api.enableTestServer': True} set
[2022-06-12 11:10:04,136] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:04,137] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:04,869] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:06,042] - [basetestcase:2754] INFO - create 2016.0 to default documents...
[2022-06-12 11:10:06,070] - [data_helper:308] INFO - creating direct client 127.0.0.1:12000 default
[2022-06-12 11:10:08,615] - [basetestcase:2767] INFO - LOAD IS FINISHED
[2022-06-12 11:10:08,648] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:08,648] - [newtuq:94] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-06-12 11:10:08,648] - [basetestcase:833] INFO - sleep for 30 secs.  ...
[2022-06-12 11:10:38,676] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-06-12 11:10:38,680] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:10:38,684] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:10:38,704] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 18.384825ms
[2022-06-12 11:10:38,704] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b20b6c49-4b9e-4624-95cc-4049f44646a4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '18.384825ms', 'executionTime': '18.312587ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-06-12 11:10:38,704] - [tuq_helper:758] INFO - Create primary index
[2022-06-12 11:10:38,708] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-06-12 11:10:38,711] - [rest_client:4170] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-06-12 11:10:39,093] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 380.034224ms
[2022-06-12 11:10:39,094] - [tuq_helper:760] INFO - Check if index is online
[2022-06-12 11:10:39,099] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:10:39,109] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:10:39,122] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.366385ms
[2022-06-12 11:10:39,144] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:39,144] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:39,942] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:39,960] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:39,964] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-06-12 11:10:39,977] - [rest_client:2069] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-06-12 11:10:39,977] - [rest_client:1111] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-06-12 11:10:39,985] - [rest_client:2069] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-06-12 11:10:39,994] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:40,009] - [basetestcase:2645] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:40,012] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:40,012] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:40,727] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:40,736] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:10:40,736] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-06-12 11:10:40,849] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:10:40,858] - [basetestcase:2645] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:40,862] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:40,862] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:41,580] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:41,589] - [rest_client:1870] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-06-12 11:10:41,589] - [remote_util:3334] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-06-12 11:10:41,706] - [remote_util:3383] INFO - command executed successfully with Administrator
[2022-06-12 11:10:41,717] - [basetestcase:2645] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-06-12 11:10:41,718] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-06-12 11:10:41,718] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:10:41,721] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-06-12 11:10:41,725] - [rest_client:4170] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-06-12 11:10:41,732] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.665872ms
[2022-06-12 11:10:41,735] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-06-12 11:10:41,738] - [rest_client:4170] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-06-12 11:10:41,789] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.890841ms
[2022-06-12 11:10:41,805] - [basetestcase:587] INFO - ------- Cluster statistics -------
[2022-06-12 11:10:41,805] - [basetestcase:589] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.179408305256474, 'mem_free': 12886601728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-06-12 11:10:41,806] - [basetestcase:590] INFO - --- End of cluster statistics ---
[2022-06-12 11:10:41,811] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:41,811] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:42,531] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:42,536] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:42,536] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:43,257] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:43,262] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:43,262] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:44,397] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:44,405] - [remote_util:302] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-06-12 11:10:44,405] - [remote_util:338] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-06-12 11:10:45,652] - [remote_util:3658] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-06-12 11:10:53,754] - [basetestcase:705] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-06-12 11:10:53,783] - [bucket_helper:127] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-06-12 11:10:54,076] - [bucket_helper:218] INFO - waiting for bucket deletion to complete....
[2022-06-12 11:10:54,079] - [rest_client:140] INFO - node 127.0.0.1 existing buckets : []
[2022-06-12 11:10:54,080] - [bucket_helper:150] INFO - deleted bucket : default from 127.0.0.1
[2022-06-12 11:10:54,085] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:10:54,090] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:10:54,095] - [bucket_helper:152] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-06-12 11:10:54,096] - [basetestcase:724] INFO - Removing user 'clientuser'...
[2022-06-12 11:10:54,106] - [rest_client:1075] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-06-12 11:10:54,107] - [basetestcase:728] INFO - b'"User was not found."'
[2022-06-12 11:10:54,110] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:10:54,122] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-06-12 11:10:54,122] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:10:54,125] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-06-12 11:10:54,128] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-06-12 11:10:54,129] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:10:54,133] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-06-12 11:10:54,136] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-06-12 11:10:54,136] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:10:54,139] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-06-12 11:10:54,142] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-06-12 11:10:54,142] - [rest_client:44] INFO - -->is_ns_server_running?
[2022-06-12 11:10:54,145] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-06-12 11:10:54,145] - [basetestcase:747] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-06-12 11:10:54,146] - [basetestcase:759] INFO - closing all ssh connections
[2022-06-12 11:10:54,146] - [basetestcase:764] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 76.887s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91:  4684 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests	40.887s
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests	285.277s
--- FAIL: TestRangeWithConcurrentAddMuts (10.67s)
--- FAIL: TestRangeWithConcurrentDelMuts (24.36s)
--- FAIL: TestScanWithConcurrentIndexOps (59.90s)
--- FAIL: TestConcurrentScans_SameIndex (59.56s)
--- FAIL: TestConcurrentScans_MultipleIndexes (60.98s)
--- FAIL: TestMutationsWithMultipleIndexBuilds (64.63s)
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
Version: versions-12.06.2022-09.30.cfg
Build Log: make-12.06.2022-09.30.log
Server Log: logs-12.06.2022-09.30.tar.gz

Finished