Building

Started building at 2022/09/12 00:32:05
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 12.09.2022-00.32
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at 0760e59, changes since last good build: 
fatal: Invalid revision range 52396bd6b64d8340b403a6ca18517986458493d9..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable c4d64115] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 4c99dbcd] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 50e91333] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 170851 at refs/changes/51/170851/9
Total 7 (delta 3), reused 3 (delta 3)
[unstable 0e48447] MB-48747: Fix deadlock between shard shutdown & sharedLSSCleaner
 Author: saptarshi.sen 
 Date: Tue Feb 15 12:47:45 2022 -0800
 5 files changed, 84 insertions(+), 6 deletions(-)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable 98c1414] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 3906576] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 7f024b0] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable 0bb90f7] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable 7b7e038] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable 4eaa49b] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Pulling in uncommitted change 179485 at refs/changes/85/179485/1
Total 48 (delta 39), reused 46 (delta 39)
[unstable 1c5995a] MB-52020: Export fields in struct ShardTransferStatistics
 Author: saptarshi.sen 
 Date: Thu Sep 1 15:43:53 2022 -0700
 2 files changed, 41 insertions(+), 41 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/12 01:23:11
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Sep 12 01:28:36 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 01:32:28 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 01:33:12 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 12 01:34:34 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 12 01:35:21 b" 'stop_on_failure': 'True'}"
Sep 12 01:35:21 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 01:35:21 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_1'}"
Sep 12 01:35:21 b'-->result: '
Sep 12 01:35:21 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 12 01:35:21 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 01:35:21 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_2'}"
Sep 12 01:35:21 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:35:21 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 12 01:35:21 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_3'}"
Sep 12 01:35:21 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:35:21 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:35:21 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 12 01:35:21 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_4'}"
Sep 12 01:35:21 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:35:21 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:35:21 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 01:45:16 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 12 01:46:01 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 12 01:49:46 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 01:53:44 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 01:56:27 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 01:56:27 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_5'}"
Sep 12 01:56:27 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 12 01:56:27 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 12 01:56:27 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_6'}"
Sep 12 01:56:27 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 01:56:27 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 12 01:56:27 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_7'}"
Sep 12 01:56:27 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 12 01:56:27 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 12 01:56:27 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-23-33/test_8'}"
Sep 12 01:56:27 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 01:56:27 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 01:56:27 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 12 01:56:27 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 12 01:56:28 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 01:56:28 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 01:56:28 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 12 01:56:28 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 12 01:56:28 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 12 01:56:28 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 12 01:56:28 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 12 01:56:28 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 187: go: command not found
/home/buildbot/bin/dotest: line 190: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-12 01:59:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:20 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:20 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:20 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 01:59:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 01:59:29 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:41 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:41 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:42 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:43 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 01:59:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-12 01:59:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 2172219392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 01:59:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 01:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 01:59:50 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 1995616256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 01:59:50 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 01:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 01:59:51 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1875193856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 01:59:51 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 01:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 01:59:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1775460352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 01:59:52 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 01:59:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 01:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 01:59:53 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 01:59:53 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 01:59:53 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-12 01:59:54 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 01:59:59 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 01:59:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 01:59:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 01:59:59 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 02:00:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:00:01 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:00:01 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 49.6649709071604, 'mem_free': 13403766784, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 44.97433567047119
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 02:00:05 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 02:00:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 02:00:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:10 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:10 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:11 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 02:00:11 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:00:13 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 02:00:13 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:13 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 02:00:13 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 147.146623ms
2022-09-12 02:00:43 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'fc2cfcb4-257f-472c-8527-102e92c42cc3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '147.146623ms', 'executionTime': '147.022518ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 02:00:43 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 695.745172ms
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.677456ms
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:44 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.670579ms
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.269076ms
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.495013891220955, 'mem_free': 13366005760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:45 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:46 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:50 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 02:00:50 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 02:00:51 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:51 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 02:00:52 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.027s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:52 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.62973896900229, 'mem_free': 13335441408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 02:00:57 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 02:00:57 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 02:00:58 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13335441408, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 02:00:59 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13329793024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:00:59 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13330071552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:00 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13329838080, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 02:01:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 02:01:01 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 02:01:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 02:01:07 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 02:01:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 02:01:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 02:01:07 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 02:01:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:01:08 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:01:09 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.66338369293046, 'mem_free': 13317439488, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.390891790390015
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 02:01:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 02:01:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 02:01:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:01:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:01:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:01:20 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 02:01:20 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 02:01:21 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 02:01:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:01:21 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 02:01:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 02:01:51 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 02:01:51 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.552481ms
2022-09-12 02:01:52 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '776740ef-0f8f-4c07-aac1-e09bbb51513f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.552481ms', 'executionTime': '68.407311ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 744.450461ms
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:01:52 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.851606ms
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 02:01:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:01:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee222dd7417f3246c29bf70095825821dfjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-12 02:01:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee222dd7417f3246c29bf70095825821dfjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-12 02:01:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 78.076235ms
2022-09-12 02:01:54 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee222dd7417f3246c29bf70095825821dfjob_title) USING GSI
2022-09-12 02:01:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee222dd7417f3246c29bf70095825821dfjob_title) USING GSI
2022-09-12 02:01:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee222dd7417f3246c29bf70095825821dfjob_title%29+USING+GSI
2022-09-12 02:01:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.761364ms
2022-09-12 02:01:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee222dd7417f3246c29bf70095825821dfjob_title'
2022-09-12 02:01:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee222dd7417f3246c29bf70095825821dfjob_title%27
2022-09-12 02:01:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.529613ms
2022-09-12 02:01:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee222dd7417f3246c29bf70095825821dfjob_title'
2022-09-12 02:01:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee222dd7417f3246c29bf70095825821dfjob_title%27
2022-09-12 02:01:57 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.193583ms
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.708685ms
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'a72f4850-c0ba-4fec-b8c7-4f2cb55e7139', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee222dd7417f3246c29bf70095825821dfjob_title', 'index_id': '3c20e568f51b43f1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.708685ms', 'executionTime': '2.582101ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:01:58 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-12 02:01:58 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 138.878055ms
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-12 02:01:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-12 02:02:00 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:02:00 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee222dd7417f3246c29bf70095825821dfjob_title'
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee222dd7417f3246c29bf70095825821dfjob_title%27
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.210269ms
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee222dd7417f3246c29bf70095825821dfjob_title ON default USING GSI
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee222dd7417f3246c29bf70095825821dfjob_title+ON+default+USING+GSI
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 42.44393ms
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee222dd7417f3246c29bf70095825821dfjob_title'
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee222dd7417f3246c29bf70095825821dfjob_title%27
2022-09-12 02:02:01 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.043683ms
2022-09-12 02:02:01 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '9ce1ec7e-6b27-4c8d-a8ba-6b9537a8625d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.043683ms', 'executionTime': '5.924481ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 02:02:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:02:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.415777ms
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 02:02:02 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.346643ms
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.70596329099938, 'mem_free': 13213904896, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:03 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 02:02:04 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 02:02:04 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 02:02:08 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 02:02:08 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 02:02:09 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 02:02:09 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.913s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_2'}
[2022-09-12 02:02:10,044] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:10,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:10,327] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:10,359] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:02:10,441] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:02:10,442] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-12 02:02:10,442] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:02:11,073] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:02:11,100] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13237596160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:02:11,126] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:02:11,126] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:02:11,126] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:02:11,155] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:02:11,195] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:02:11,195] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:02:11,222] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:02:11,223] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:02:11,224] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:02:11,224] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:02:11,271] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:02:11,274] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:11,274] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:11,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:11,534] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:02:11,601] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:02:11,602] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:02:11,635] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:11,662] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:11,693] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:02:11,818] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:02:11,844] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13197422592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:02:11,869] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:02:11,897] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:02:11,898] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:02:11,948] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:02:11,951] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:11,951] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:12,217] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:12,218] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:02:12,286] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:02:12,287] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:02:12,316] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:12,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:12,370] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:02:12,482] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:02:12,509] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13203275776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:02:12,534] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:02:12,561] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:02:12,561] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:02:12,612] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:02:12,617] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:12,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:12,885] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:12,886] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:02:12,954] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:02:12,955] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:02:12,984] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:13,011] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:13,040] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:02:13,154] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:02:13,182] - [task:165] INFO -  {'uptime': '160', 'memoryTotal': 15466930176, 'memoryFree': 13238292480, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:02:13,208] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:02:13,236] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:02:13,236] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:02:13,290] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:02:13,293] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:13,293] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:13,555] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:13,556] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:02:13,625] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:02:13,626] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:02:13,655] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:13,684] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:02:13,713] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:02:13,803] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:02:14,336] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:02:19,342] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:02:19,429] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:02:19,433] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:19,433] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:19,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:19,725] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:02:19,797] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:02:19,798] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:02:19,798] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:02:20,925] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:02:20,981] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 02:02:20,982] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:02:21,973] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:02:22,287] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:02:22,574] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:02:22,576] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:02:22,628] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:22,628] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:22,884] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:22,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:22,888] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:23,143] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:23,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:23,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:23,396] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:23,400] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:23,401] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:23,741] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:27,632] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:02:27,632] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.82091671948353, 'mem_free': 13232353280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:02:27,632] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:02:27,633] - [basetestcase:467] INFO - Time to execute basesetup : 17.590462684631348
[2022-09-12 02:02:27,685] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:02:27,686] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:02:27,744] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:02:27,744] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:02:27,800] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:02:27,801] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:02:27,862] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:02:27,862] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:02:27,918] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:02:27,980] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:02:27,980] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:02:27,981] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:02:32,992] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:02:32,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:02:32,996] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:02:33,277] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:02:34,223] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:02:34,387] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:02:36,992] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:02:37,013] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 02:02:37,670] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:02:37,671] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:02:37,671] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:03:07,689] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:03:07,718] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:03:07,743] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:03:07,808] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.160149ms
[2022-09-12 02:03:07,809] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e78b2e4c-58ff-40d9-b863-f1378e4519e4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.160149ms', 'executionTime': '64.091389ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:03:07,809] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:03:07,835] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:03:07,862] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:03:08,576] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 712.140632ms
[2022-09-12 02:03:08,577] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:03:08,639] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:03:08,675] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:03:08,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.452886ms
[2022-09-12 02:03:08,876] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:08,914] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:03:08,928] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:03:08,929] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:03:08,947] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:03:09,029] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:09,839] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef22c93b896d14372a670c8eecfcd11f9job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:03:09,870] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef22c93b896d14372a670c8eecfcd11f9job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:03:09,927] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.848061ms
[2022-09-12 02:03:09,928] - [base_gsi:282] INFO - BUILD INDEX on default(employeef22c93b896d14372a670c8eecfcd11f9job_title) USING GSI
[2022-09-12 02:03:10,957] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeef22c93b896d14372a670c8eecfcd11f9job_title) USING GSI
[2022-09-12 02:03:10,982] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeef22c93b896d14372a670c8eecfcd11f9job_title%29+USING+GSI
[2022-09-12 02:03:11,004] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.977831ms
[2022-09-12 02:03:12,035] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef22c93b896d14372a670c8eecfcd11f9job_title'
[2022-09-12 02:03:12,060] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef22c93b896d14372a670c8eecfcd11f9job_title%27
[2022-09-12 02:03:12,069] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.406065ms
[2022-09-12 02:03:13,099] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef22c93b896d14372a670c8eecfcd11f9job_title'
[2022-09-12 02:03:13,125] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef22c93b896d14372a670c8eecfcd11f9job_title%27
[2022-09-12 02:03:13,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.735555ms
[2022-09-12 02:03:13,590] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 02:03:13,763] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:03:14,685] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:03:15,138] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:15,171] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:03:15,197] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 02:03:15,201] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.28834ms
[2022-09-12 02:03:15,201] - [task:3246] INFO - {'requestID': 'c4743d54-ac20-4eab-9e13-9daec19b78cd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef22c93b896d14372a670c8eecfcd11f9job_title', 'index_id': '18ec9ff72ea9ec10', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.28834ms', 'executionTime': '2.17186ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 02:03:15,202] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:15,202] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:15,202] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:03:15,203] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:03:15,203] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 02:03:15,203] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:03:15,204] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:03:15,204] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:03:15,204] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:03:16,203] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:16,234] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:03:16,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 02:03:16,387] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 116.047929ms
[2022-09-12 02:03:16,387] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:03:16,388] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:03:17,168] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:17,169] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:03:18,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef22c93b896d14372a670c8eecfcd11f9job_title'
[2022-09-12 02:03:18,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef22c93b896d14372a670c8eecfcd11f9job_title%27
[2022-09-12 02:03:18,237] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.889644ms
[2022-09-12 02:03:18,263] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef22c93b896d14372a670c8eecfcd11f9job_title ON default USING GSI
[2022-09-12 02:03:18,288] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef22c93b896d14372a670c8eecfcd11f9job_title+ON+default+USING+GSI
[2022-09-12 02:03:18,328] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.475032ms
[2022-09-12 02:03:18,364] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef22c93b896d14372a670c8eecfcd11f9job_title'
[2022-09-12 02:03:18,392] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef22c93b896d14372a670c8eecfcd11f9job_title%27
[2022-09-12 02:03:18,400] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.769847ms
[2022-09-12 02:03:18,400] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '48571306-b22e-4854-9743-14d46cc3c225', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.769847ms', 'executionTime': '5.652365ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:03:18,502] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:18,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:18,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:18,851] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:18,906] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:03:18,907] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:03:18,976] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:03:19,032] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:19,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:19,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:19,390] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:19,452] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:03:19,452] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:03:19,519] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:03:19,576] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:19,577] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:03:19,577] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:03:19,603] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:03:19,628] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:03:19,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.574371ms
[2022-09-12 02:03:19,664] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:03:19,689] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:03:19,757] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.320337ms
[2022-09-12 02:03:19,826] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:03:19,826] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.723263094846456, 'mem_free': 13141741568, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:03:19,826] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:03:19,830] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:19,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:20,176] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:20,181] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:20,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:20,512] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:20,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:20,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:20,995] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:21,004] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:21,005] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:21,579] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:25,652] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:03:25,794] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:03:26,427] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:03:26,454] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:03:26,454] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:03:26,508] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:03:26,561] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:03:26,614] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:03:26,615] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:03:26,695] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:03:26,696] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:03:26,721] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:26,849] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:03:26,849] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:26,876] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:03:26,901] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:03:26,902] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:26,943] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:03:26,968] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:03:26,968] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:26,995] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:03:27,021] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:03:27,021] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:27,048] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:03:27,048] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:03:27,049] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:03:27,049] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.058s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_3'}
[2022-09-12 02:03:27,150] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:27,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:27,465] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:27,496] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:03:27,575] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:03:27,575] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-12 02:03:27,576] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:03:28,181] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:03:28,208] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13111857152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:03:28,234] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:03:28,234] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:03:28,234] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:03:28,271] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:03:28,305] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:03:28,306] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:03:28,334] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:03:28,335] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:03:28,335] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:03:28,335] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:03:28,385] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:03:28,388] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:28,388] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:28,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:28,694] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:03:28,767] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:03:28,768] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:03:28,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:28,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:28,856] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:03:28,978] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:03:29,010] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13180358656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:03:29,036] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:03:29,065] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:03:29,065] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:03:29,118] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:03:29,121] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:29,121] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:29,456] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:29,457] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:03:29,532] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:03:29,533] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:03:29,564] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:29,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:29,622] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:03:29,746] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:03:29,774] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13180076032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:03:29,802] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:03:29,832] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:03:29,832] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:03:29,886] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:03:29,892] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:29,892] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:30,214] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:30,215] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:03:30,295] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:03:30,297] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:03:30,326] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:30,353] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:30,382] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:03:30,506] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:03:30,533] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13179002880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:03:30,559] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:03:30,588] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:03:30,588] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:03:30,640] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:03:30,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:30,643] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:30,978] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:30,980] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:03:31,062] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:03:31,063] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:03:31,096] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:31,123] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:03:31,152] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:03:31,243] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:03:31,779] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:03:36,782] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:03:36,868] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:03:36,876] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:36,876] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:37,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:37,240] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:03:37,318] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:03:37,320] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:03:37,320] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:03:38,384] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:03:38,442] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 02:03:38,443] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:03:39,339] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:03:39,651] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:03:39,944] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:03:39,947] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:03:40,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:40,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:40,331] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:40,335] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:40,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:40,634] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:40,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:40,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:40,937] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:40,943] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:40,943] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:41,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:45,618] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:03:45,619] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.98904018741236, 'mem_free': 13181784064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:03:45,619] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:03:45,619] - [basetestcase:467] INFO - Time to execute basesetup : 18.470906734466553
[2022-09-12 02:03:45,669] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:03:45,669] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:03:45,723] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:03:45,723] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:03:45,777] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:03:45,777] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:03:45,830] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:03:45,830] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:03:45,881] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:45,946] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:03:45,946] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:03:45,946] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:03:50,957] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:03:50,961] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:03:50,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:03:51,268] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:03:52,225] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:03:52,395] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:03:54,998] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:03:55,013] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 02:03:55,853] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:03:55,854] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:03:55,854] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:04:25,881] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:04:25,910] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:04:25,935] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:04:26,002] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.039489ms
[2022-09-12 02:04:26,002] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2bd2b35e-3bc4-4d82-9e88-7606b9e5b121', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.039489ms', 'executionTime': '64.90742ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:04:26,002] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:04:26,028] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:04:26,053] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:04:26,734] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 678.906529ms
[2022-09-12 02:04:26,734] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:04:26,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:04:26,830] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:04:26,837] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.980077ms
[2022-09-12 02:04:27,031] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:04:27,077] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:04:27,092] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:04:27,094] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:04:27,114] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:04:27,197] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:04:27,994] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee8fea98693eca464c9980f3d1b699d94djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:04:28,019] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee8fea98693eca464c9980f3d1b699d94djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:04:28,067] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.934069ms
[2022-09-12 02:04:28,067] - [base_gsi:282] INFO - BUILD INDEX on default(employee8fea98693eca464c9980f3d1b699d94djob_title) USING GSI
[2022-09-12 02:04:29,098] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee8fea98693eca464c9980f3d1b699d94djob_title) USING GSI
[2022-09-12 02:04:29,124] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee8fea98693eca464c9980f3d1b699d94djob_title%29+USING+GSI
[2022-09-12 02:04:29,147] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.460722ms
[2022-09-12 02:04:30,179] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8fea98693eca464c9980f3d1b699d94djob_title'
[2022-09-12 02:04:30,205] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8fea98693eca464c9980f3d1b699d94djob_title%27
[2022-09-12 02:04:30,215] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.454789ms
[2022-09-12 02:04:31,245] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8fea98693eca464c9980f3d1b699d94djob_title'
[2022-09-12 02:04:31,270] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8fea98693eca464c9980f3d1b699d94djob_title%27
[2022-09-12 02:04:31,277] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.920629ms
[2022-09-12 02:04:31,650] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 02:04:31,902] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:04:32,689] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:04:33,279] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:33,312] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:04:33,340] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 02:04:33,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.20972ms
[2022-09-12 02:04:33,344] - [task:3246] INFO - {'requestID': '0ca74206-3e2e-4d24-ad35-ed573d2cbeb2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8fea98693eca464c9980f3d1b699d94djob_title', 'index_id': '9c47a30026649b21', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.20972ms', 'executionTime': '2.137102ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 02:04:33,344] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:33,344] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:33,345] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:04:33,345] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:04:33,346] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 02:04:33,346] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:04:33,346] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:04:33,346] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:04:33,346] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:04:34,346] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:34,376] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:04:34,402] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 02:04:34,530] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 118.439175ms
[2022-09-12 02:04:34,531] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:04:34,531] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:04:35,317] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:35,318] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:04:36,350] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8fea98693eca464c9980f3d1b699d94djob_title'
[2022-09-12 02:04:36,377] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8fea98693eca464c9980f3d1b699d94djob_title%27
[2022-09-12 02:04:36,386] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.985687ms
[2022-09-12 02:04:36,413] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee8fea98693eca464c9980f3d1b699d94djob_title ON default USING GSI
[2022-09-12 02:04:36,440] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee8fea98693eca464c9980f3d1b699d94djob_title+ON+default+USING+GSI
[2022-09-12 02:04:36,484] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.269742ms
[2022-09-12 02:04:36,523] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee8fea98693eca464c9980f3d1b699d94djob_title'
[2022-09-12 02:04:36,549] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8fea98693eca464c9980f3d1b699d94djob_title%27
[2022-09-12 02:04:36,556] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.327459ms
[2022-09-12 02:04:36,556] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '673380a6-bb36-4fe9-b028-702fe81ffd05', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.327459ms', 'executionTime': '5.241504ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:04:36,664] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:04:36,668] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:36,668] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:37,054] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:37,110] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:04:37,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:04:37,183] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:04:37,237] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:04:37,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:37,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:37,624] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:37,687] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:04:37,688] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:04:37,758] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:04:37,814] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:04:37,814] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:04:37,814] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:04:37,840] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:04:37,866] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:04:37,875] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.140218ms
[2022-09-12 02:04:37,901] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:04:37,927] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:04:37,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.384259ms
[2022-09-12 02:04:38,040] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:04:38,040] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.08173510548304, 'mem_free': 13051031552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:04:38,041] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:04:38,045] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:38,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:38,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:38,443] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:38,443] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:38,980] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:38,988] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:38,988] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:39,667] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:39,678] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:39,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:40,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:44,552] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:04:44,697] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:04:45,895] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:04:45,922] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:04:45,922] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:04:45,982] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:04:46,035] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:04:46,088] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:04:46,089] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:04:46,166] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:04:46,167] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:04:46,193] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:46,321] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:04:46,322] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:46,348] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:04:46,373] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:04:46,373] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:46,399] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:04:46,424] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:04:46,425] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:46,451] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:04:46,477] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:04:46,477] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:46,503] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:04:46,504] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:04:46,504] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:04:46,504] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.407s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_4'}
[2022-09-12 02:04:46,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:46,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:46,959] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:46,989] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:04:47,070] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:04:47,070] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-12 02:04:47,071] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:04:47,632] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:04:47,658] - [task:165] INFO -  {'uptime': '323', 'memoryTotal': 15466930176, 'memoryFree': 13043105792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:04:47,683] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:04:47,683] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:04:47,684] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:04:47,718] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:04:47,753] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:04:47,753] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:04:47,780] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:04:47,781] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:04:47,781] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:04:47,782] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:04:47,831] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:04:47,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:47,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:48,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:48,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:04:48,272] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:04:48,273] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:04:48,302] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:48,327] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:48,355] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:04:48,478] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:04:48,505] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13138694144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:04:48,530] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:04:48,557] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:04:48,558] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:04:48,609] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:04:48,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:48,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:48,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:48,971] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:04:49,054] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:04:49,055] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:04:49,084] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:49,110] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:49,137] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:04:49,252] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:04:49,280] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13139070976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:04:49,314] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:04:49,342] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:04:49,342] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:04:49,392] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:04:49,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:49,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:49,775] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:49,776] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:04:49,869] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:04:49,871] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:04:49,902] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:49,931] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:49,960] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:04:50,082] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:04:50,109] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13138685952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:04:50,135] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:04:50,164] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:04:50,164] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:04:50,219] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:04:50,222] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:50,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:50,586] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:50,587] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:04:50,667] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:04:50,668] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:04:50,698] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:50,723] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:04:50,752] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:04:50,840] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:04:51,466] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:04:56,471] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:04:56,555] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:04:56,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:56,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:04:56,936] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:04:56,937] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:04:57,021] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:04:57,022] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:04:57,022] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:04:57,992] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:04:58,045] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:04:58,045] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:04:58,942] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:04:59,255] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:04:59,547] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:04:59,550] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:04:59,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:04:59,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:00,005] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:00,009] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:00,009] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:00,405] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:00,409] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:00,410] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:00,991] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:01,004] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:01,005] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:01,701] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:06,199] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:05:06,200] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.20355883351712, 'mem_free': 13136904192, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:05:06,200] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:05:06,200] - [basetestcase:467] INFO - Time to execute basesetup : 19.599512338638306
[2022-09-12 02:05:06,252] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:05:06,252] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:05:06,306] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:05:06,306] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:05:06,360] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:05:06,360] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:05:06,416] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:05:06,416] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:05:06,471] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:06,542] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:05:06,542] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:05:06,542] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:05:11,550] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:05:11,553] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:11,553] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:11,950] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:12,935] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:05:13,181] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:05:15,666] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:05:15,681] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 02:05:16,356] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:16,356] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:05:16,356] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:05:46,386] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:05:46,416] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:05:46,442] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:05:46,509] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.733664ms
[2022-09-12 02:05:46,510] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3df9fb51-3af5-4310-a9fc-6a67ee9676d8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.733664ms', 'executionTime': '65.666524ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:05:46,510] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:05:46,536] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:05:46,561] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:05:47,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 732.213229ms
[2022-09-12 02:05:47,296] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:05:47,357] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:05:47,397] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:05:47,406] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.546192ms
[2022-09-12 02:05:47,592] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:47,627] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:05:47,641] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:05:47,642] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:05:47,656] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:05:47,728] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:48,555] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef5a5e159be9340c99aa9c8f6b0b21608job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:05:48,584] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef5a5e159be9340c99aa9c8f6b0b21608job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:05:48,646] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.311647ms
[2022-09-12 02:05:48,647] - [base_gsi:282] INFO - BUILD INDEX on default(employeef5a5e159be9340c99aa9c8f6b0b21608job_title) USING GSI
[2022-09-12 02:05:49,677] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeef5a5e159be9340c99aa9c8f6b0b21608job_title) USING GSI
[2022-09-12 02:05:49,702] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeef5a5e159be9340c99aa9c8f6b0b21608job_title%29+USING+GSI
[2022-09-12 02:05:49,723] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.548259ms
[2022-09-12 02:05:50,754] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef5a5e159be9340c99aa9c8f6b0b21608job_title'
[2022-09-12 02:05:50,780] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef5a5e159be9340c99aa9c8f6b0b21608job_title%27
[2022-09-12 02:05:50,790] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.797902ms
[2022-09-12 02:05:51,823] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef5a5e159be9340c99aa9c8f6b0b21608job_title'
[2022-09-12 02:05:51,850] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef5a5e159be9340c99aa9c8f6b0b21608job_title%27
[2022-09-12 02:05:51,858] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.314911ms
[2022-09-12 02:05:52,338] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 02:05:52,507] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:05:53,372] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:05:53,557] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-12 02:05:53,558] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:05:53,734] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-12 02:05:53,735] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-12 02:05:53,735] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3871679953, 0, b'')
[2022-09-12 02:05:53,861] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:53,888] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:05:53,913] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 02:05:53,917] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.371681ms
[2022-09-12 02:05:53,917] - [task:3246] INFO - {'requestID': '605ce49d-5c67-4bda-86ad-8867b8fbd35b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef5a5e159be9340c99aa9c8f6b0b21608job_title', 'index_id': 'b7d5fed89ab3430b', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.371681ms', 'executionTime': '2.304344ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 02:05:53,917] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:53,918] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:53,918] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:05:53,918] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:05:53,919] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 02:05:53,919] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:05:53,919] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:05:53,919] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:05:53,919] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:05:54,919] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:54,948] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:05:54,973] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 02:05:55,099] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 117.110325ms
[2022-09-12 02:05:55,099] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:05:55,100] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:05:55,892] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:55,892] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:05:56,924] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef5a5e159be9340c99aa9c8f6b0b21608job_title'
[2022-09-12 02:05:56,950] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef5a5e159be9340c99aa9c8f6b0b21608job_title%27
[2022-09-12 02:05:56,962] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.310589ms
[2022-09-12 02:05:56,990] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef5a5e159be9340c99aa9c8f6b0b21608job_title ON default USING GSI
[2022-09-12 02:05:57,015] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef5a5e159be9340c99aa9c8f6b0b21608job_title+ON+default+USING+GSI
[2022-09-12 02:05:57,060] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.594067ms
[2022-09-12 02:05:57,103] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef5a5e159be9340c99aa9c8f6b0b21608job_title'
[2022-09-12 02:05:57,132] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef5a5e159be9340c99aa9c8f6b0b21608job_title%27
[2022-09-12 02:05:57,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.456418ms
[2022-09-12 02:05:57,143] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fed0975b-dc65-4a61-a58f-f56e8be865b0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '9.456418ms', 'executionTime': '9.380623ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:05:57,248] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:57,251] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:57,251] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:57,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:57,739] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:05:57,740] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:05:57,814] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:05:57,869] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:57,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:57,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:58,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:58,393] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:05:58,394] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:05:58,471] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:05:58,527] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:05:58,527] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:05:58,527] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:05:58,552] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:05:58,578] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:05:58,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.24324ms
[2022-09-12 02:05:58,614] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:05:58,639] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:05:58,694] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.658034ms
[2022-09-12 02:05:58,763] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:05:58,763] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.701139038290215, 'mem_free': 13013508096, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:05:58,764] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:05:58,768] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:58,768] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:59,206] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:59,211] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:59,211] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:05:59,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:05:59,915] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:05:59,916] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:00,722] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:00,730] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:00,730] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:01,538] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:06,161] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:06:06,305] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:06:07,494] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:06:07,522] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:06:07,523] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:06:07,583] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:06:07,636] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:06:07,690] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:06:07,691] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:06:07,769] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:06:07,770] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:06:07,795] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:07,926] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:06:07,926] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:07,953] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:06:07,979] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:06:07,979] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:08,006] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:06:08,032] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:06:08,032] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:08,064] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:06:08,089] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:06:08,089] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:08,116] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:06:08,116] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:06:08,116] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:06:08,117] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 81.567s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_5'}
[2022-09-12 02:06:08,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:08,217] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:08,640] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:08,671] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:06:08,754] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:06:08,754] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-12 02:06:08,755] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:06:09,243] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:06:09,270] - [task:165] INFO -  {'uptime': '405', 'memoryTotal': 15466930176, 'memoryFree': 13096288256, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:06:09,296] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:06:09,297] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:06:09,297] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:06:09,328] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:06:09,365] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:06:09,366] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:06:09,394] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:06:09,395] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:06:09,395] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:06:09,396] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:06:09,446] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:06:09,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:09,449] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:09,903] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:09,904] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:06:09,995] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:06:09,997] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:06:10,026] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:10,052] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:10,082] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:06:10,210] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:06:10,237] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13096931328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:06:10,262] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:06:10,291] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:06:10,291] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:06:10,345] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:06:10,348] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:10,348] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:10,806] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:10,807] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:06:10,905] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:06:10,906] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:06:10,936] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:10,963] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:10,993] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:06:11,113] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:06:11,142] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13097811968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:06:11,170] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:06:11,199] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:06:11,200] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:06:11,251] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:06:11,255] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:11,255] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:11,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:11,684] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:06:11,780] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:06:11,781] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:06:11,810] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:11,836] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:11,864] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:06:11,977] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:06:12,004] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13096505344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:06:12,030] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:06:12,057] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:06:12,058] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:06:12,108] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:06:12,111] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:12,111] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:12,527] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:12,528] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:06:12,615] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:06:12,616] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:06:12,646] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:12,671] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:06:12,699] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:06:12,789] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:06:13,362] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:06:18,368] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:06:18,451] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:06:18,455] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:18,455] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:18,879] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:18,880] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:06:18,969] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:06:18,970] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:06:18,971] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:06:19,917] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:06:19,971] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:06:19,971] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:06:20,866] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:06:21,173] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:06:21,449] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:06:21,452] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:06:21,511] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:21,511] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:21,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:21,938] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:21,938] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:22,335] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:22,339] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:22,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:23,017] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:23,025] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:23,025] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:23,762] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:28,604] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:06:28,604] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.92951234247566, 'mem_free': 13104467968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:06:28,604] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:06:28,605] - [basetestcase:467] INFO - Time to execute basesetup : 20.390575647354126
[2022-09-12 02:06:28,655] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:06:28,656] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:06:28,712] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:06:28,712] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:06:28,767] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:06:28,767] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:06:28,824] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:06:28,824] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:06:28,878] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:06:28,947] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:06:28,947] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:06:28,947] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:06:33,957] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:06:33,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:06:33,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:06:34,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:06:35,421] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:06:35,679] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:06:37,982] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:06:37,998] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 02:06:39,060] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:06:39,060] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:06:39,060] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:07:09,081] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:07:09,111] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:07:09,136] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:07:09,205] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.728867ms
[2022-09-12 02:07:09,205] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8ccd0670-9d9b-4dc4-af8b-e0c0282e5671', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.728867ms', 'executionTime': '66.662183ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:07:09,206] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:07:09,231] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:07:09,256] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:07:09,908] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 649.915164ms
[2022-09-12 02:07:09,908] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:07:09,990] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:07:10,042] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:07:10,053] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.429732ms
[2022-09-12 02:07:10,252] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:10,288] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:07:10,308] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:07:10,309] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:07:10,321] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:07:10,387] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:11,208] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:07:11,233] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:07:11,291] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.655495ms
[2022-09-12 02:07:11,292] - [base_gsi:282] INFO - BUILD INDEX on default(employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title) USING GSI
[2022-09-12 02:07:12,323] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title) USING GSI
[2022-09-12 02:07:12,348] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%29+USING+GSI
[2022-09-12 02:07:12,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.052749ms
[2022-09-12 02:07:13,403] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title'
[2022-09-12 02:07:13,428] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%27
[2022-09-12 02:07:13,437] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.757431ms
[2022-09-12 02:07:14,466] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title'
[2022-09-12 02:07:14,492] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%27
[2022-09-12 02:07:14,500] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.727122ms
[2022-09-12 02:07:14,982] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:07:15,157] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:07:17,871] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:07:18,504] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:18,537] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:07:18,562] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 02:07:18,567] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.736682ms
[2022-09-12 02:07:18,567] - [task:3246] INFO - {'requestID': '670e3d7a-808d-4bf3-8850-249dc92813f1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title', 'index_id': '3271f041ed85f1f6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.736682ms', 'executionTime': '2.625609ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 02:07:18,567] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:18,567] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:18,568] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:07:18,568] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:07:18,569] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 02:07:18,569] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:07:18,569] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:07:18,569] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:07:18,569] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:07:19,568] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:19,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:07:19,628] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 02:07:19,793] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 148.808839ms
[2022-09-12 02:07:19,793] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:07:19,794] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:07:21,363] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:21,363] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:07:22,394] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title'
[2022-09-12 02:07:22,419] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%27
[2022-09-12 02:07:22,429] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.081372ms
[2022-09-12 02:07:22,454] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title ON default USING GSI
[2022-09-12 02:07:22,480] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title+ON+default+USING+GSI
[2022-09-12 02:07:22,527] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.148135ms
[2022-09-12 02:07:22,561] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title'
[2022-09-12 02:07:22,590] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeea4adc2c68f5c400b90a9c0f4a3f554cajob_title%27
[2022-09-12 02:07:22,596] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.295066ms
[2022-09-12 02:07:22,597] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ee668bea-3ad8-4b19-a3b5-d9a1443e5ee9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.295066ms', 'executionTime': '5.231687ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:07:22,702] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:22,705] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:22,705] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:23,214] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:23,269] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:07:23,269] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:07:23,367] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:07:23,421] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:23,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:23,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:23,922] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:23,982] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:07:23,982] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:07:24,070] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:07:24,125] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:24,125] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:07:24,125] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:07:24,150] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:07:24,175] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:07:24,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.550538ms
[2022-09-12 02:07:24,210] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:07:24,235] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:07:24,285] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.12743ms
[2022-09-12 02:07:24,353] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:07:24,353] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.35073396721874, 'mem_free': 12921143296, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:07:24,353] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:07:24,357] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:24,357] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:24,860] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:24,865] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:24,865] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:25,372] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:25,377] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:25,377] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:26,041] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:26,049] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:26,049] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:26,993] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:33,193] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:07:33,334] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:07:34,485] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:07:34,513] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:07:34,514] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:07:34,573] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:07:34,627] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:07:34,680] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:07:34,681] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:07:34,761] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:07:34,762] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:07:34,787] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:34,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:07:34,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:34,943] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:07:34,971] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:07:34,971] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:34,997] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:07:35,024] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:07:35,025] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:35,051] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:07:35,077] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:07:35,077] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:35,103] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:07:35,104] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:07:35,104] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:07:35,104] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.941s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_6'}
[2022-09-12 02:07:35,207] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:35,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:35,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:35,725] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:07:35,804] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:07:35,804] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-12 02:07:35,805] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:07:36,233] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:07:36,260] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13015801856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:07:36,286] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:07:36,286] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:07:36,286] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:07:36,326] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:07:36,360] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:07:36,361] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:07:36,390] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:07:36,391] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:07:36,391] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:07:36,391] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:07:36,440] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:07:36,443] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:36,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:36,979] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:36,980] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:07:37,092] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:07:37,093] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:07:37,123] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:37,154] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:37,183] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:07:37,304] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:07:37,331] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13016731648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:07:37,356] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:07:37,386] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:07:37,387] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:07:37,438] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:07:37,441] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:37,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:37,925] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:37,926] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:07:38,027] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:07:38,029] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:07:38,058] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:38,083] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:38,111] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:07:38,227] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:07:38,254] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13016399872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:07:38,280] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:07:38,307] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:07:38,308] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:07:38,357] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:07:38,362] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:38,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:38,860] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:38,861] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:07:38,959] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:07:38,960] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:07:38,990] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:39,015] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:39,044] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:07:39,160] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:07:39,188] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13016014848, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:07:39,213] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:07:39,240] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:07:39,240] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:07:39,289] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:07:39,293] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:39,293] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:39,797] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:39,799] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:07:39,899] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:07:39,901] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:07:39,940] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:39,966] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:07:39,999] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:07:40,094] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:07:40,617] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:07:45,622] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:07:45,704] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:07:45,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:45,709] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:46,222] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:46,223] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:07:46,330] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:07:46,331] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:07:46,332] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:07:47,248] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:07:47,301] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:07:47,301] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:07:48,205] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:07:48,510] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:07:48,768] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:07:48,771] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:07:48,829] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:48,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:49,315] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:49,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:49,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:49,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:49,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:49,868] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:50,758] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:50,767] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:07:50,767] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:07:51,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:07:57,198] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:07:57,199] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.83098365475776, 'mem_free': 13019987968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:07:57,199] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:07:57,199] - [basetestcase:467] INFO - Time to execute basesetup : 21.993958711624146
[2022-09-12 02:07:57,250] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:07:57,250] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:07:57,302] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:07:57,302] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:07:57,357] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:07:57,357] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:07:57,410] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:07:57,410] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:07:57,462] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:07:57,524] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:07:57,526] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:07:57,526] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:08:02,538] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:08:02,541] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:08:02,542] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:08:03,058] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:08:04,078] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:08:04,341] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:08:06,623] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:08:06,640] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 02:08:08,015] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:08,015] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:08:08,015] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:08:38,040] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:08:38,069] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:08:38,095] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:08:38,169] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 72.090427ms
[2022-09-12 02:08:38,170] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '354b1766-b71c-4c7c-877b-413fda2b3e36', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '72.090427ms', 'executionTime': '72.029972ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:08:38,170] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:08:38,196] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:08:38,222] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:08:38,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 666.148608ms
[2022-09-12 02:08:38,890] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:08:38,951] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:08:38,985] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:08:38,994] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.102266ms
[2022-09-12 02:08:39,218] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:39,255] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:08:39,268] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:08:39,268] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:08:39,283] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:08:39,349] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:40,179] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec93732ad65d746c2ba6e946d15734576job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:08:40,206] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec93732ad65d746c2ba6e946d15734576job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:08:40,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 88.41988ms
[2022-09-12 02:08:40,297] - [base_gsi:282] INFO - BUILD INDEX on default(employeec93732ad65d746c2ba6e946d15734576job_title) USING GSI
[2022-09-12 02:08:41,328] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec93732ad65d746c2ba6e946d15734576job_title) USING GSI
[2022-09-12 02:08:41,359] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec93732ad65d746c2ba6e946d15734576job_title%29+USING+GSI
[2022-09-12 02:08:41,382] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.977826ms
[2022-09-12 02:08:42,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec93732ad65d746c2ba6e946d15734576job_title'
[2022-09-12 02:08:42,439] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec93732ad65d746c2ba6e946d15734576job_title%27
[2022-09-12 02:08:42,449] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.506227ms
[2022-09-12 02:08:43,480] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec93732ad65d746c2ba6e946d15734576job_title'
[2022-09-12 02:08:43,507] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec93732ad65d746c2ba6e946d15734576job_title%27
[2022-09-12 02:08:43,517] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.423355ms
[2022-09-12 02:08:44,028] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 02:08:44,192] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:08:45,342] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:08:45,970] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 02:08:46,150] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:08:47,469] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:08:47,860] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:08:48,033] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:08:51,702] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:08:52,538] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:52,570] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:08:52,597] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 02:08:52,601] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.335963ms
[2022-09-12 02:08:52,602] - [task:3246] INFO - {'requestID': 'e948695d-e3e7-4f5b-a5e1-4803216b9365', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec93732ad65d746c2ba6e946d15734576job_title', 'index_id': '66c08dd35e80c0d5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.335963ms', 'executionTime': '3.271577ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 02:08:52,602] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:52,602] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:52,603] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:08:52,603] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:08:52,603] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 02:08:52,604] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:08:52,604] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:08:52,604] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:08:52,604] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 02:08:53,604] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:53,634] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 02:08:53,659] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 02:08:54,038] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 155.648807ms
[2022-09-12 02:08:54,038] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:08:54,039] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:08:55,588] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:55,588] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 02:08:56,619] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec93732ad65d746c2ba6e946d15734576job_title'
[2022-09-12 02:08:56,644] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec93732ad65d746c2ba6e946d15734576job_title%27
[2022-09-12 02:08:56,653] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.708245ms
[2022-09-12 02:08:56,678] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec93732ad65d746c2ba6e946d15734576job_title ON default USING GSI
[2022-09-12 02:08:56,705] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec93732ad65d746c2ba6e946d15734576job_title+ON+default+USING+GSI
[2022-09-12 02:08:56,745] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.717621ms
[2022-09-12 02:08:56,779] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec93732ad65d746c2ba6e946d15734576job_title'
[2022-09-12 02:08:56,809] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec93732ad65d746c2ba6e946d15734576job_title%27
[2022-09-12 02:08:56,816] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.932169ms
[2022-09-12 02:08:56,816] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9a83f9a8-7a15-41e3-8667-fd2aed330117', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.932169ms', 'executionTime': '5.859931ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:08:56,921] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:56,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:08:56,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:08:57,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:08:57,652] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:08:57,652] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:08:57,755] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:08:57,809] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:57,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:08:57,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:08:58,441] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:08:58,495] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:08:58,496] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:08:58,600] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:08:58,657] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:08:58,657] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:08:58,657] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:08:58,683] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:08:58,708] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:08:58,718] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.012892ms
[2022-09-12 02:08:58,745] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:08:58,770] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:08:58,823] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.789153ms
[2022-09-12 02:08:58,896] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:08:58,896] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.2942298097334, 'mem_free': 12790517760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:08:58,896] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:08:58,900] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:08:58,900] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:08:59,542] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:08:59,547] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:08:59,547] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:00,638] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:00,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:00,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:01,777] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:01,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:01,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:02,940] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:09,473] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:09:09,623] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:09:10,807] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:09:10,835] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:09:10,835] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:09:10,900] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:09:10,953] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:09:11,006] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:09:11,007] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:09:11,085] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:09:11,086] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:09:11,111] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:11,241] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:09:11,241] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:11,268] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:09:11,293] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:09:11,293] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:11,319] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:09:11,344] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:09:11,344] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:11,370] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:09:11,399] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:09:11,399] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:11,425] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:09:11,426] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 02:09:11,426] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:09:11,427] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 96.272s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_7'}
[2022-09-12 02:09:11,556] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:11,557] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:12,144] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:12,178] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:09:12,258] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:09:12,258] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-12 02:09:12,259] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:09:12,583] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:09:12,609] - [task:165] INFO -  {'uptime': '588', 'memoryTotal': 15466930176, 'memoryFree': 12962553856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:09:12,635] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:09:12,635] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:09:12,636] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:09:12,670] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:09:12,705] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:09:12,705] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:09:12,733] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:09:12,734] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:09:12,734] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:09:12,734] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:09:12,785] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:09:12,787] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:12,788] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:13,416] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:13,418] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:09:13,537] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:09:13,539] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:09:13,569] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:13,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:13,627] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:09:13,753] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:09:13,781] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12963917824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:09:13,812] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:09:13,842] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:09:13,842] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:09:13,892] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:09:13,895] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:13,895] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:14,516] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:14,518] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:09:14,638] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:09:14,639] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:09:14,670] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:14,699] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:14,729] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:09:14,849] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:09:14,876] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12969570304, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:09:14,902] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:09:14,930] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:09:14,930] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:09:14,981] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:09:14,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:14,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:15,587] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:15,588] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:09:15,701] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:09:15,702] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:09:15,731] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:15,756] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:15,784] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:09:15,902] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:09:15,929] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12976750592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:09:15,955] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:09:15,982] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:09:15,983] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:09:16,033] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:09:16,036] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:16,036] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:16,635] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:16,636] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:09:16,754] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:09:16,755] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:09:16,784] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:16,810] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:09:16,839] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:09:16,924] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:09:17,580] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:09:22,581] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:09:22,665] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:09:22,670] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:22,671] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:23,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:23,285] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:09:23,403] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:09:23,404] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:09:23,405] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:09:24,041] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:09:24,096] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:09:24,097] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:09:25,007] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:09:25,313] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:09:25,577] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:09:25,580] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 02:09:25,633] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:25,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:26,270] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:26,274] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:26,275] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:27,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:27,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:27,172] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:28,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:28,268] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:28,268] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:29,321] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:35,911] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:09:35,911] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.7316810964466, 'mem_free': 12976062464, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:09:35,911] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:09:35,912] - [basetestcase:467] INFO - Time to execute basesetup : 24.357524156570435
[2022-09-12 02:09:35,963] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:09:35,963] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:09:36,017] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:09:36,017] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:09:36,071] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:09:36,071] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:09:36,125] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:09:36,125] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:09:36,177] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:09:36,236] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:09:36,236] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:09:36,236] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:09:41,245] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:09:41,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:09:41,249] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:09:41,891] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:09:42,932] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:09:43,108] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:09:45,319] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:09:45,398] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:09:45,398] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:09:45,399] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:10:15,423] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:10:15,453] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:10:15,478] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:10:15,545] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.91233ms
[2022-09-12 02:10:15,546] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '62131bb1-8f0f-49be-a816-07f38076b5e9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.91233ms', 'executionTime': '64.84023ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:10:15,546] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:10:15,571] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:10:15,596] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:10:16,285] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 687.788092ms
[2022-09-12 02:10:16,285] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:10:16,368] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:10:16,400] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:10:16,408] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.249695ms
[2022-09-12 02:10:16,613] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:10:16,647] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:10:16,666] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:10:16,667] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:10:16,679] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:10:16,680] - [base_gsi:326] INFO - []
[2022-09-12 02:10:17,570] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeebdc909dc207442228bd4fb422b44a3cdjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:10:17,596] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeebdc909dc207442228bd4fb422b44a3cdjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:10:17,645] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.383653ms
[2022-09-12 02:10:17,680] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeebdc909dc207442228bd4fb422b44a3cdjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:10:17,706] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeebdc909dc207442228bd4fb422b44a3cdjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:10:17,763] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.236398ms
[2022-09-12 02:10:17,763] - [base_gsi:282] INFO - BUILD INDEX on default(employeebdc909dc207442228bd4fb422b44a3cdjob_title,employeebdc909dc207442228bd4fb422b44a3cdjoin_yr) USING GSI
[2022-09-12 02:10:18,794] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeebdc909dc207442228bd4fb422b44a3cdjob_title,employeebdc909dc207442228bd4fb422b44a3cdjoin_yr) USING GSI
[2022-09-12 02:10:18,822] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeebdc909dc207442228bd4fb422b44a3cdjob_title%2Cemployeebdc909dc207442228bd4fb422b44a3cdjoin_yr%29+USING+GSI
[2022-09-12 02:10:18,862] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.725263ms
[2022-09-12 02:10:19,894] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjob_title'
[2022-09-12 02:10:19,920] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjob_title%27
[2022-09-12 02:10:19,929] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.620762ms
[2022-09-12 02:10:20,962] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjob_title'
[2022-09-12 02:10:20,988] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjob_title%27
[2022-09-12 02:10:20,997] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.899329ms
[2022-09-12 02:10:21,028] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjoin_yr'
[2022-09-12 02:10:21,055] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjoin_yr%27
[2022-09-12 02:10:21,059] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.804602ms
[2022-09-12 02:10:22,091] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjob_title'
[2022-09-12 02:10:22,116] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjob_title%27
[2022-09-12 02:10:22,127] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.406087ms
[2022-09-12 02:10:22,153] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeebdc909dc207442228bd4fb422b44a3cdjob_title ON default USING GSI
[2022-09-12 02:10:22,177] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeebdc909dc207442228bd4fb422b44a3cdjob_title+ON+default+USING+GSI
[2022-09-12 02:10:22,217] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.565312ms
[2022-09-12 02:10:22,255] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjoin_yr'
[2022-09-12 02:10:22,280] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjoin_yr%27
[2022-09-12 02:10:22,288] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.97857ms
[2022-09-12 02:10:22,315] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeebdc909dc207442228bd4fb422b44a3cdjoin_yr ON default USING GSI
[2022-09-12 02:10:22,340] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeebdc909dc207442228bd4fb422b44a3cdjoin_yr+ON+default+USING+GSI
[2022-09-12 02:10:22,386] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.574163ms
[2022-09-12 02:10:22,420] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjob_title'
[2022-09-12 02:10:22,446] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjob_title%27
[2022-09-12 02:10:22,452] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.368602ms
[2022-09-12 02:10:22,453] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c521cb63-ba0b-4f89-b12b-8759469338cf', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.368602ms', 'executionTime': '5.291387ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:10:22,481] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebdc909dc207442228bd4fb422b44a3cdjoin_yr'
[2022-09-12 02:10:22,505] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebdc909dc207442228bd4fb422b44a3cdjoin_yr%27
[2022-09-12 02:10:22,508] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.418332ms
[2022-09-12 02:10:22,509] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ec7a3cac-1a78-4182-b7aa-91c6d9b3c6a5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.418332ms', 'executionTime': '1.357387ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:10:22,608] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:10:22,612] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:22,612] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:23,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:23,303] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:10:23,304] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:10:23,405] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:10:23,466] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:10:23,469] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:23,470] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:24,058] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:24,117] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:10:24,118] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:10:24,219] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:10:24,274] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:10:24,274] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:10:24,275] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:10:24,301] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:10:24,327] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:10:24,337] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.880649ms
[2022-09-12 02:10:24,362] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:10:24,387] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:10:24,437] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.334741ms
[2022-09-12 02:10:24,506] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:10:24,506] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.5994428908077, 'mem_free': 12860133376, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:10:24,506] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:10:24,511] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:24,511] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:25,107] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:25,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:25,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:25,705] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:25,711] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:25,711] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:26,539] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:26,547] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:26,547] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:27,604] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:34,677] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-12 02:10:34,836] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:10:35,679] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:10:35,707] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:10:35,708] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:10:35,768] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:10:35,822] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:10:35,877] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:10:35,878] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:10:35,971] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:10:35,973] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:10:36,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:36,141] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:10:36,142] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:36,170] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:10:36,198] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:10:36,198] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:36,225] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:10:36,251] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:10:36,251] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:36,279] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:10:36,306] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:10:36,306] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:36,333] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:10:36,333] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 02:10:36,333] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:10:36,334] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.834s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_8'}
[2022-09-12 02:10:36,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:36,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:37,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:37,065] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:10:37,148] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:10:37,148] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-12 02:10:37,149] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:10:37,441] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:10:37,467] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12975939584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:10:37,493] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:10:37,493] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:10:37,494] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:10:37,527] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:10:37,560] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:10:37,560] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:10:37,588] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:10:37,589] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:10:37,589] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:10:37,589] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:10:37,636] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:10:37,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:37,640] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:38,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:38,211] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:10:38,323] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:10:38,324] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:10:38,354] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:38,379] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:38,408] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:10:38,528] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:10:38,555] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 12974723072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:10:38,580] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:10:38,607] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:10:38,608] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:10:38,657] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:10:38,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:38,660] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:39,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:39,247] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:10:39,362] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:10:39,363] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:10:39,392] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:39,417] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:39,445] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:10:39,558] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:10:39,585] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 12973985792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:10:39,611] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:10:39,639] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:10:39,639] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:10:39,691] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:10:39,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:39,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:40,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:40,284] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:10:40,408] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:10:40,409] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:10:40,439] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:40,465] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:40,493] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:10:40,607] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:10:40,633] - [task:165] INFO -  {'uptime': '667', 'memoryTotal': 15466930176, 'memoryFree': 12974587904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:10:40,658] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:10:40,686] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:10:40,686] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:10:40,742] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:10:40,745] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:40,745] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:41,345] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:41,346] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:10:41,464] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:10:41,465] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:10:41,495] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:41,521] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:10:41,550] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:10:41,637] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:10:42,170] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:10:47,175] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:10:47,260] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:10:47,265] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:47,265] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:47,860] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:47,861] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:10:47,984] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:10:47,985] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:10:47,985] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:10:48,792] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:10:48,845] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:10:48,846] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:10:49,755] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:10:50,069] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:10:50,517] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:10:50,521] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 02:10:50,574] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:50,574] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:51,168] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:51,173] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:51,173] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:51,982] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:51,991] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:51,991] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:53,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:10:53,069] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:10:53,069] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:10:54,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:00,591] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:11:00,591] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.85180819009395, 'mem_free': 12976611328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:11:00,591] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:11:00,591] - [basetestcase:467] INFO - Time to execute basesetup : 24.17899441719055
[2022-09-12 02:11:00,646] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:11:00,646] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:11:00,700] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:11:00,700] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:11:00,757] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:11:00,757] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:11:00,814] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:11:00,815] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:11:00,871] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:00,931] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:11:00,931] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:11:00,931] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:11:05,940] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:11:05,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:05,945] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:06,537] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:07,583] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:11:07,752] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:11:09,918] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:11:10,004] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:10,004] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:11:10,004] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:11:40,016] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:11:40,045] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:11:40,070] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:11:40,139] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.033096ms
[2022-09-12 02:11:40,140] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '331eff25-2c0c-4ce5-b684-6684db162e1a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.033096ms', 'executionTime': '66.961444ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:11:40,140] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:11:40,166] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:11:40,193] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:11:40,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 695.9985ms
[2022-09-12 02:11:40,891] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:11:40,963] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:11:40,997] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:11:41,007] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.636128ms
[2022-09-12 02:11:41,250] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:41,287] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:11:41,303] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:11:41,304] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:11:41,315] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:11:41,316] - [base_gsi:326] INFO - []
[2022-09-12 02:11:42,210] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6bf45c33fe98403299cb67711f820b56job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:11:42,236] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:11:42,300] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.170418ms
[2022-09-12 02:11:42,301] - [base_gsi:282] INFO - BUILD INDEX on default(employee6bf45c33fe98403299cb67711f820b56job_title_join_yr) USING GSI
[2022-09-12 02:11:43,333] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee6bf45c33fe98403299cb67711f820b56job_title_join_yr) USING GSI
[2022-09-12 02:11:43,360] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%29+USING+GSI
[2022-09-12 02:11:43,383] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.410178ms
[2022-09-12 02:11:44,414] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6bf45c33fe98403299cb67711f820b56job_title_join_yr'
[2022-09-12 02:11:44,440] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%27
[2022-09-12 02:11:44,451] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.395478ms
[2022-09-12 02:11:45,481] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6bf45c33fe98403299cb67711f820b56job_title_join_yr'
[2022-09-12 02:11:45,506] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%27
[2022-09-12 02:11:45,516] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.8118ms
[2022-09-12 02:11:46,548] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6bf45c33fe98403299cb67711f820b56job_title_join_yr'
[2022-09-12 02:11:46,573] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%27
[2022-09-12 02:11:46,582] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.96946ms
[2022-09-12 02:11:46,611] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee6bf45c33fe98403299cb67711f820b56job_title_join_yr ON default USING GSI
[2022-09-12 02:11:46,637] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee6bf45c33fe98403299cb67711f820b56job_title_join_yr+ON+default+USING+GSI
[2022-09-12 02:11:46,685] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.459614ms
[2022-09-12 02:11:46,724] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6bf45c33fe98403299cb67711f820b56job_title_join_yr'
[2022-09-12 02:11:46,751] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6bf45c33fe98403299cb67711f820b56job_title_join_yr%27
[2022-09-12 02:11:46,759] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.387781ms
[2022-09-12 02:11:46,759] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b5545287-737e-4713-85a7-3a9489825588', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.387781ms', 'executionTime': '6.319038ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:11:46,870] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:46,873] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:46,873] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:47,508] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:47,563] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:11:47,563] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:11:47,663] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:11:47,719] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:47,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:47,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:48,358] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:48,421] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:11:48,421] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:11:48,528] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:11:48,587] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:11:48,588] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:11:48,588] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:11:48,616] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:11:48,645] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:11:48,655] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.12314ms
[2022-09-12 02:11:48,683] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:11:48,711] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:11:48,766] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.257875ms
[2022-09-12 02:11:48,835] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:11:48,835] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.738745640166909, 'mem_free': 12854173696, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:11:48,835] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:11:48,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:48,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:49,478] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:49,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:49,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:50,480] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:50,488] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:50,488] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:51,585] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:51,593] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:11:51,593] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:11:52,718] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:11:59,120] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-12 02:11:59,263] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:12:00,438] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:12:00,466] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:12:00,466] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:12:00,530] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:12:00,584] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:12:00,639] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:12:00,640] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:12:00,721] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:12:00,722] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:12:00,748] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:00,877] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:12:00,877] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:00,904] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:12:00,930] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:12:00,931] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:00,957] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:12:00,986] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:12:00,986] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:01,015] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:12:01,041] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:12:01,041] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:01,068] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:12:01,068] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 02:12:01,068] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:12:01,069] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.711s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_9'}
[2022-09-12 02:12:01,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:01,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:01,806] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:01,839] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:12:01,920] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:12:01,920] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-12 02:12:01,921] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:12:02,184] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:12:02,212] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 12966608896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:12:02,238] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:12:02,239] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:12:02,239] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:12:02,275] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:12:02,309] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:12:02,309] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:12:02,338] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:12:02,339] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:12:02,339] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:12:02,339] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:12:02,389] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:12:02,392] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:02,392] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:02,999] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:03,001] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:12:03,124] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:12:03,125] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:12:03,154] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:03,180] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:03,208] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:12:03,325] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:12:03,351] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 12969127936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:12:03,377] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:12:03,407] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:12:03,408] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:12:03,460] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:12:03,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:03,463] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:04,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:04,122] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:12:04,249] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:12:04,250] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:12:04,282] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:04,310] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:04,341] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:12:04,470] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:12:04,499] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 12969992192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:12:04,526] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:12:04,556] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:12:04,557] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:12:04,610] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:12:04,615] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:04,616] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:05,275] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:05,277] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:12:05,405] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:12:05,406] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:12:05,438] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:05,469] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:05,499] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:12:05,625] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:12:05,653] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 12970258432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:12:05,681] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:12:05,710] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:12:05,711] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:12:05,762] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:12:05,766] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:05,766] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:06,389] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:06,391] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:12:06,515] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:12:06,516] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:12:06,546] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:06,573] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:12:06,602] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:12:06,695] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:12:07,226] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:12:12,231] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:12:12,313] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:12:12,318] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:12,318] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:12,924] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:12,925] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:12:13,048] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:12:13,049] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:12:13,049] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:12:13,853] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:12:13,910] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 02:12:13,911] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:12:14,978] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:12:15,317] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:12:15,607] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:12:15,610] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 02:12:15,663] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:15,663] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:16,271] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:16,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:16,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:17,133] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:17,146] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:17,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:18,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:18,270] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:18,270] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:19,321] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:25,883] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:12:25,883] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.34853796104925, 'mem_free': 12965752832, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:12:25,883] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:12:25,883] - [basetestcase:467] INFO - Time to execute basesetup : 24.72913146018982
[2022-09-12 02:12:25,937] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:12:25,937] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:12:25,992] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:12:25,993] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:12:26,047] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:12:26,047] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:12:26,100] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:12:26,100] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:12:26,151] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:12:26,216] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:12:26,216] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:12:26,216] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:12:31,226] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:12:31,229] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:12:31,230] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:12:31,887] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:12:33,051] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:12:33,233] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:12:35,305] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:12:35,384] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:12:35,385] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:12:35,385] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:13:05,411] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:13:05,441] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:13:05,466] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:13:05,533] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.469396ms
[2022-09-12 02:13:05,533] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '181add08-c4d4-4f2b-9306-989547c2e6be', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.469396ms', 'executionTime': '65.399377ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:13:05,533] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:13:05,558] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:13:05,583] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:13:06,269] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 683.815795ms
[2022-09-12 02:13:06,269] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:13:06,343] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:13:06,379] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:13:06,387] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.634155ms
[2022-09-12 02:13:06,595] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:06,628] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:13:06,646] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:13:06,647] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:13:06,670] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:13:06,730] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:07,557] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee1d3b57934e9640119282bf4b6fee9348join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:13:07,582] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee1d3b57934e9640119282bf4b6fee9348join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:13:07,633] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.584143ms
[2022-09-12 02:13:07,634] - [base_gsi:282] INFO - BUILD INDEX on default(`employee1d3b57934e9640119282bf4b6fee9348join_yr`) USING GSI
[2022-09-12 02:13:08,665] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee1d3b57934e9640119282bf4b6fee9348join_yr`) USING GSI
[2022-09-12 02:13:08,691] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee1d3b57934e9640119282bf4b6fee9348join_yr%60%29+USING+GSI
[2022-09-12 02:13:08,720] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.163744ms
[2022-09-12 02:13:08,762] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1d3b57934e9640119282bf4b6fee9348join_yr'
[2022-09-12 02:13:08,799] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1d3b57934e9640119282bf4b6fee9348join_yr%27
[2022-09-12 02:13:08,808] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.948898ms
[2022-09-12 02:13:09,840] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1d3b57934e9640119282bf4b6fee9348join_yr'
[2022-09-12 02:13:09,868] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1d3b57934e9640119282bf4b6fee9348join_yr%27
[2022-09-12 02:13:09,880] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.132235ms
[2022-09-12 02:13:10,910] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1d3b57934e9640119282bf4b6fee9348join_yr'
[2022-09-12 02:13:10,936] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1d3b57934e9640119282bf4b6fee9348join_yr%27
[2022-09-12 02:13:10,944] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.314171ms
[2022-09-12 02:13:11,975] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:13:12,002] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 02:13:12,007] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.783791ms
[2022-09-12 02:13:12,008] - [base_gsi:504] INFO - {'requestID': 'af766e6d-0dd5-4764-8a9b-0b254a6ab655', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee1d3b57934e9640119282bf4b6fee9348join_yr', 'index_id': '63d35a1fd0c7c519', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.783791ms', 'executionTime': '3.715388ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 02:13:12,008] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:13:12,009] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:13:12,009] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:13:12,009] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:13:12,010] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:13:12,011] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:13:12,011] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:13:12,011] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:13:12,073] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:13:12,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:13:12,136] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 02:13:12,310] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 162.173676ms
[2022-09-12 02:13:12,311] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:13:12,311] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:13:14,450] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:14,477] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1d3b57934e9640119282bf4b6fee9348join_yr'
[2022-09-12 02:13:14,504] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1d3b57934e9640119282bf4b6fee9348join_yr%27
[2022-09-12 02:13:14,510] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.418643ms
[2022-09-12 02:13:14,510] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c867fca9-6c3b-490d-bb36-e3a8afb5c0bc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.418643ms', 'executionTime': '4.34821ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:13:14,613] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:14,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:14,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:15,344] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:15,404] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:13:15,404] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:13:15,513] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:13:15,574] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:15,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:15,577] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:16,249] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:16,304] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:13:16,304] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:13:16,412] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:13:16,469] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:16,470] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:13:16,470] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:13:16,495] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:13:16,521] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:13:16,526] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.331703ms
[2022-09-12 02:13:16,526] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '16ef0dfe-995b-4614-ae44-42c07f41be8a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.331703ms', 'executionTime': '3.273177ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:13:16,579] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:13:16,579] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.63196775601037, 'mem_free': 12832022528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:13:16,579] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:13:16,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:16,584] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:17,215] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:17,220] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:17,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:17,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:17,842] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:17,842] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:18,845] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:18,857] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:18,857] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:19,978] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:27,180] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-12 02:13:27,312] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:13:27,368] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:13:27,422] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:13:27,482] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:13:27,482] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:13:27,563] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:13:27,565] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:13:27,590] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:27,719] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:13:27,719] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:27,745] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:13:27,770] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:13:27,770] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:27,796] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:13:27,821] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:13:27,822] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:27,848] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:13:27,873] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:13:27,873] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:27,903] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:13:27,903] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 02:13:27,903] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:13:27,904] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.802s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_10'}
[2022-09-12 02:13:27,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:27,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:28,668] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:28,701] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:13:28,780] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:13:28,780] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-12 02:13:28,781] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:13:29,012] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:13:29,043] - [task:165] INFO -  {'uptime': '842', 'memoryTotal': 15466930176, 'memoryFree': 12844941312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:13:29,069] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:13:29,069] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:13:29,069] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:13:29,106] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:13:29,140] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:13:29,140] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:13:29,169] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:13:29,170] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:13:29,170] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:13:29,170] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:13:29,217] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:13:29,220] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:29,221] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:29,860] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:29,862] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:13:29,985] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:13:29,986] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:13:30,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:30,043] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:30,070] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:13:30,195] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:13:30,221] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12978974720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:13:30,247] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:13:30,274] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:13:30,276] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:13:30,326] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:13:30,332] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:30,332] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:30,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:30,972] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:13:31,093] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:13:31,094] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:13:31,124] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:31,149] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:31,177] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:13:31,294] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:13:31,321] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12978872320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:13:31,345] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:13:31,373] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:13:31,373] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:13:31,423] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:13:31,427] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:31,427] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:32,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:32,064] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:13:32,184] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:13:32,185] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:13:32,216] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:32,242] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:32,271] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:13:32,385] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:13:32,412] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12979568640, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:13:32,438] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:13:32,465] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:13:32,465] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:13:32,519] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:13:32,522] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:32,522] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:33,141] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:33,143] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:13:33,261] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:13:33,263] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:13:33,291] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:33,317] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:13:33,345] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:13:33,436] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:13:34,098] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:13:39,100] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:13:39,188] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:13:39,198] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:39,198] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:39,850] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:39,851] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:13:39,983] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:13:39,984] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:13:39,985] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:13:40,575] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:13:40,627] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:13:40,628] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:13:41,527] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:13:41,849] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:13:42,137] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:13:42,141] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 02:13:42,202] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:42,202] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:42,864] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:42,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:42,869] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:43,767] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:43,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:43,776] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:44,890] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:44,902] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:44,902] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:46,042] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:52,728] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:13:52,729] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.94034640032088, 'mem_free': 12977098752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:13:52,729] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:13:52,729] - [basetestcase:467] INFO - Time to execute basesetup : 24.74562954902649
[2022-09-12 02:13:52,783] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:13:52,783] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:13:52,836] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:13:52,837] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:13:52,889] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:13:52,889] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:13:52,941] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:13:52,942] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:13:52,991] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:13:53,053] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:13:53,054] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:13:53,054] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:13:58,064] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:13:58,067] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:13:58,067] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:13:58,764] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:13:59,929] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:14:00,108] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:14:03,183] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:14:03,260] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:03,260] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:14:03,261] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:14:33,285] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:14:33,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:14:33,340] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:14:33,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.80403ms
[2022-09-12 02:14:33,411] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3fa8c4e6-717a-439d-84ba-4b8cf98a3d8b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.80403ms', 'executionTime': '68.718252ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:14:33,411] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:14:33,437] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:14:33,461] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:14:34,126] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 662.629703ms
[2022-09-12 02:14:34,128] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:14:34,187] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:14:34,230] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:14:34,238] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.11698ms
[2022-09-12 02:14:34,411] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:34,445] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:14:34,467] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:14:34,468] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:14:34,489] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:14:34,566] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:35,382] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee11141cae8cbb43a4a57f687f85325fc5join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:14:35,407] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee11141cae8cbb43a4a57f687f85325fc5join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:14:35,456] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.666313ms
[2022-09-12 02:14:35,456] - [base_gsi:282] INFO - BUILD INDEX on default(`employee11141cae8cbb43a4a57f687f85325fc5join_yr`) USING GSI
[2022-09-12 02:14:36,487] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee11141cae8cbb43a4a57f687f85325fc5join_yr`) USING GSI
[2022-09-12 02:14:36,514] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee11141cae8cbb43a4a57f687f85325fc5join_yr%60%29+USING+GSI
[2022-09-12 02:14:36,536] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.851933ms
[2022-09-12 02:14:36,588] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee11141cae8cbb43a4a57f687f85325fc5join_yr'
[2022-09-12 02:14:36,619] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee11141cae8cbb43a4a57f687f85325fc5join_yr%27
[2022-09-12 02:14:36,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.55502ms
[2022-09-12 02:14:37,659] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee11141cae8cbb43a4a57f687f85325fc5join_yr'
[2022-09-12 02:14:37,687] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee11141cae8cbb43a4a57f687f85325fc5join_yr%27
[2022-09-12 02:14:37,697] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.626735ms
[2022-09-12 02:14:38,727] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee11141cae8cbb43a4a57f687f85325fc5join_yr'
[2022-09-12 02:14:38,751] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee11141cae8cbb43a4a57f687f85325fc5join_yr%27
[2022-09-12 02:14:38,760] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.716512ms
[2022-09-12 02:14:39,791] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:14:39,817] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 02:14:39,821] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.718213ms
[2022-09-12 02:14:39,821] - [base_gsi:504] INFO - {'requestID': 'e84f49da-0a36-4f0a-b26b-4f4df6be8762', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee11141cae8cbb43a4a57f687f85325fc5join_yr', 'index_id': 'deceb12300e75a30', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.718213ms', 'executionTime': '2.649617ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 02:14:39,822] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:14:39,822] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:39,822] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:14:39,822] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:14:39,823] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:39,823] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:39,823] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:39,823] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:39,885] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:39,928] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:14:39,956] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 02:14:40,145] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 176.715874ms
[2022-09-12 02:14:40,145] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:14:40,146] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:14:41,361] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-12 02:14:41,416] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-12 02:14:41,445] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:14:41,471] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 02:14:41,475] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.002369ms
[2022-09-12 02:14:41,476] - [base_gsi:504] INFO - {'requestID': 'f9ffecf6-7b8f-4f86-9548-9e860c880d47', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee11141cae8cbb43a4a57f687f85325fc5join_yr', 'index_id': 'deceb12300e75a30', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.002369ms', 'executionTime': '2.923448ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 02:14:41,477] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:14:41,477] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:41,477] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:14:41,477] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:14:41,477] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:41,478] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:41,478] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:41,478] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:14:41,532] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:14:41,570] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:14:41,596] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 02:14:41,686] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 81.604269ms
[2022-09-12 02:14:41,687] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:14:41,687] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:14:42,909] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee11141cae8cbb43a4a57f687f85325fc5join_yr ON default USING GSI
[2022-09-12 02:14:42,935] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee11141cae8cbb43a4a57f687f85325fc5join_yr+ON+default+USING+GSI
[2022-09-12 02:14:42,981] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.120117ms
[2022-09-12 02:14:43,021] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee11141cae8cbb43a4a57f687f85325fc5join_yr'
[2022-09-12 02:14:43,047] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee11141cae8cbb43a4a57f687f85325fc5join_yr%27
[2022-09-12 02:14:43,053] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.189193ms
[2022-09-12 02:14:43,054] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b8d45510-0d37-43b9-9a4b-07cbe8ba418f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.189193ms', 'executionTime': '5.122786ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:14:43,154] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:43,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:43,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:43,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:43,893] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:14:43,893] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:14:44,011] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:14:44,069] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:44,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:44,072] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:44,744] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:44,798] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:14:44,799] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:14:44,909] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:14:44,965] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:14:44,965] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:14:44,965] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:14:44,992] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:14:45,019] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:14:45,027] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.243663ms
[2022-09-12 02:14:45,059] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:14:45,086] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:14:45,137] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.917964ms
[2022-09-12 02:14:45,209] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:14:45,210] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.81006341156056, 'mem_free': 12791209984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:14:45,210] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:14:45,213] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:45,213] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:45,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:45,882] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:45,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:47,052] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:47,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:47,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:48,271] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:48,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:48,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:49,436] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:56,280] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-12 02:14:56,423] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:14:57,235] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:14:57,263] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:14:57,264] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:14:57,318] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:14:57,370] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:14:57,422] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:14:57,423] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:14:57,503] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:14:57,504] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:14:57,529] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:57,668] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:14:57,669] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:57,695] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:14:57,719] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:14:57,719] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:57,745] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:14:57,771] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:14:57,771] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:57,796] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:14:57,822] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:14:57,822] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:57,853] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:14:57,854] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 02:14:57,854] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:14:57,855] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 89.923s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_11'}
[2022-09-12 02:14:57,936] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:57,937] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:58,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:58,628] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:14:58,707] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:14:58,707] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-12 02:14:58,708] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:14:58,963] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:14:58,989] - [task:165] INFO -  {'uptime': '935', 'memoryTotal': 15466930176, 'memoryFree': 12950298624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:14:59,014] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:14:59,015] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:14:59,015] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:14:59,048] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:14:59,090] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:14:59,090] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:14:59,117] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:14:59,118] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:14:59,118] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:14:59,118] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:14:59,167] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:14:59,170] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:14:59,170] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:14:59,846] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:14:59,847] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:14:59,973] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:14:59,974] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:15:00,005] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:00,033] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:00,064] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:15:00,189] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:15:00,216] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 12950437888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:15:00,243] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:15:00,271] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:15:00,272] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:15:00,323] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:15:00,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:00,327] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:01,022] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:01,023] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:15:01,154] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:15:01,155] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:15:01,183] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:01,209] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:01,237] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:15:01,352] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:15:01,378] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12950761472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:15:01,404] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:15:01,432] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:15:01,432] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:15:01,484] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:15:01,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:01,490] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:02,196] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:02,197] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:15:02,343] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:15:02,344] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:15:02,376] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:02,404] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:02,434] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:15:02,560] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:15:02,588] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 12952068096, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:15:02,615] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:15:02,645] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:15:02,646] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:15:02,699] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:15:02,701] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:02,702] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:03,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:03,431] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:15:03,562] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:15:03,563] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:15:03,593] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:03,620] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:15:03,648] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:15:03,736] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:15:04,397] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:15:09,402] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:15:09,485] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:15:09,491] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:09,492] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:10,191] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:10,192] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:15:10,322] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:15:10,323] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:15:10,324] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:15:10,895] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:15:10,948] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:15:10,948] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:15:11,854] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:15:12,175] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:15:12,455] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:15:12,459] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 02:15:12,518] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:12,519] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:13,228] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:13,233] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:13,234] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:14,258] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:14,270] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:14,270] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:15,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:15,447] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:15,447] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:16,610] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:23,678] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:15:23,679] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 28.50438771880721, 'mem_free': 12947550208, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:15:23,679] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:15:23,679] - [basetestcase:467] INFO - Time to execute basesetup : 25.745261192321777
[2022-09-12 02:15:23,731] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:15:23,732] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:15:23,786] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:15:23,786] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:15:23,838] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:15:23,838] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:15:23,891] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:15:23,891] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:15:23,941] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:15:24,007] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:15:24,012] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:15:24,012] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:15:29,020] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:15:29,024] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:15:29,024] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:15:29,706] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:15:30,898] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:15:31,080] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:15:34,104] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:15:34,184] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:15:34,185] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:15:34,185] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:16:04,215] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:16:04,244] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:16:04,269] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:16:04,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.258649ms
[2022-09-12 02:16:04,338] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '38bc4f7d-e5c8-4ece-bcf2-3e121a3c185f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.258649ms', 'executionTime': '67.201965ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:16:04,338] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:16:04,363] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:16:04,388] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:16:05,078] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 687.704929ms
[2022-09-12 02:16:05,078] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:16:05,151] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:16:05,192] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:16:05,200] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.86106ms
[2022-09-12 02:16:05,387] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:05,422] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:16:05,443] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:16:05,444] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:16:05,459] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:16:05,526] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:06,348] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5e897e594ac945de9d4f371a60f71a6fjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 02:16:06,373] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 02:16:06,426] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.184181ms
[2022-09-12 02:16:06,427] - [base_gsi:282] INFO - BUILD INDEX on default(`employee5e897e594ac945de9d4f371a60f71a6fjoin_yr`) USING GSI
[2022-09-12 02:16:07,459] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee5e897e594ac945de9d4f371a60f71a6fjoin_yr`) USING GSI
[2022-09-12 02:16:07,484] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%60%29+USING+GSI
[2022-09-12 02:16:07,509] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.265553ms
[2022-09-12 02:16:07,552] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr'
[2022-09-12 02:16:07,581] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%27
[2022-09-12 02:16:07,590] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.747993ms
[2022-09-12 02:16:08,621] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr'
[2022-09-12 02:16:08,646] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%27
[2022-09-12 02:16:08,656] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.530408ms
[2022-09-12 02:16:09,687] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr'
[2022-09-12 02:16:09,713] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%27
[2022-09-12 02:16:09,722] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.444724ms
[2022-09-12 02:16:10,754] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:16:10,781] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 02:16:10,786] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.518286ms
[2022-09-12 02:16:10,786] - [base_gsi:504] INFO - {'requestID': '67cab3d5-e717-41b8-822e-2b4ef952dd40', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr', 'index_id': '5949d324923db871', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.518286ms', 'executionTime': '2.455848ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 02:16:10,786] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 02:16:10,787] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:16:10,787] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 02:16:10,787] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 02:16:10,787] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:16:10,788] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:16:10,788] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:16:10,788] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 02:16:10,848] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 02:16:10,889] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 02:16:10,916] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 02:16:11,094] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 165.106206ms
[2022-09-12 02:16:11,095] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 02:16:11,095] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 02:16:13,482] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 02:16:15,887] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:16:15,949] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 02:16:15,950] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:16:16,821] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:16:17,137] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:16:17,430] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:16:17,435] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 02:16:19,467] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:16:19,467] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:19,494] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:16:19,522] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:16:19,522] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:19,548] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:16:19,577] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:16:19,577] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:19,603] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:16:19,631] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:16:19,631] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:19,657] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:16:19,708] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:19,734] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr'
[2022-09-12 02:16:19,760] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%27
[2022-09-12 02:16:19,808] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.771042ms
[2022-09-12 02:16:19,808] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c755c278-e957-4456-ad5a-9646db1474a7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '46.771042ms', 'executionTime': '46.702694ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:16:19,834] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5e897e594ac945de9d4f371a60f71a6fjoin_yr'
[2022-09-12 02:16:19,860] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5e897e594ac945de9d4f371a60f71a6fjoin_yr%27
[2022-09-12 02:16:19,864] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.586559ms
[2022-09-12 02:16:19,864] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd9f07e36-b2b4-4cb8-bafd-faaa4a427140', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.586559ms', 'executionTime': '1.482147ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:16:19,918] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:20,043] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:20,045] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:20,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:20,745] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:20,802] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:16:20,803] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:16:20,916] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:16:20,972] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:20,975] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:20,975] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:21,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:21,706] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:16:21,706] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:16:21,822] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:16:21,879] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:16:21,879] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:16:21,880] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:16:21,904] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:16:21,931] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:16:21,940] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.91744ms
[2022-09-12 02:16:21,940] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e4b12e3c-d7a6-451d-904b-6741978b763f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.91744ms', 'executionTime': '7.851599ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:16:21,941] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:16:21,968] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:16:21,993] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:16:21,995] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.397378ms
[2022-09-12 02:16:21,996] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '65280f44-eb23-4e95-809c-e31fbaf50f8b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.397378ms', 'executionTime': '1.328484ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:16:22,047] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:16:22,047] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.84701810980808, 'mem_free': 12933718016, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:16:22,047] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:16:22,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:22,051] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:22,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:22,813] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:22,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:23,597] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:23,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:23,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:24,831] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:24,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:24,839] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:26,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:33,581] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 02:16:33,734] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:16:34,047] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:16:34,076] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:16:34,076] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:16:34,133] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:16:34,190] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:16:34,249] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:16:34,250] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:16:34,331] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:16:34,332] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:16:34,357] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:34,488] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:16:34,488] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:34,515] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:16:34,542] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:16:34,542] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:34,570] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:16:34,597] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:16:34,597] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:34,624] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:16:34,649] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:16:34,650] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:34,677] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:16:34,677] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 02:16:34,677] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:16:34,678] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.796s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_01-59-19/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-12 02:16:34,737] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:34,737] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:35,471] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:35,504] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:16:35,585] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 02:16:35,586] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-12 02:16:35,586] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 02:16:35,764] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 02:16:35,791] - [task:165] INFO -  {'uptime': '1031', 'memoryTotal': 15466930176, 'memoryFree': 12934971392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:16:35,819] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 02:16:35,819] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 02:16:35,820] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 02:16:35,851] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 02:16:35,888] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 02:16:35,889] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 02:16:35,918] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 02:16:35,919] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 02:16:35,919] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 02:16:35,919] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 02:16:35,985] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:16:35,988] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:35,988] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:36,678] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:36,680] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:16:36,808] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:16:36,809] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:16:36,840] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:36,866] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:36,894] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:16:37,020] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 02:16:37,047] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 12935020544, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:16:37,072] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:16:37,100] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 02:16:37,100] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 02:16:37,154] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:16:37,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:37,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:37,870] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:37,871] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:16:37,999] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:16:38,001] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:16:38,031] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:38,056] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:38,084] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:16:38,200] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 02:16:38,227] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 12934316032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:16:38,256] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:16:38,284] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 02:16:38,285] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 02:16:38,335] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:16:38,339] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:38,339] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:39,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:39,036] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:16:39,170] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:16:39,171] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:16:39,201] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:39,226] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:39,253] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:16:39,366] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 02:16:39,392] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 12934889472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 02:16:39,418] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 02:16:39,445] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 02:16:39,445] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 02:16:39,496] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 02:16:39,499] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:39,499] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:40,220] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:40,221] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:16:40,358] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:16:40,359] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:16:40,393] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:40,420] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 02:16:40,449] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 02:16:40,541] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 02:16:41,078] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:16:46,081] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 02:16:46,166] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 02:16:46,170] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:46,170] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:46,891] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:46,892] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 02:16:47,032] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 02:16:47,033] - [remote_util:5249] INFO - b'ok'
[2022-09-12 02:16:47,033] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 02:16:47,682] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 02:16:47,737] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 02:16:47,737] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 02:16:48,663] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:16:48,989] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:16:49,493] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 02:16:49,496] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-12 02:16:49,551] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:49,551] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:50,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:50,287] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:50,287] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:51,363] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:51,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:51,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:52,667] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:16:52,675] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:16:52,675] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:16:53,938] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:01,288] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:17:01,289] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.66711984169672, 'mem_free': 12928790528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:17:01,289] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:17:01,289] - [basetestcase:467] INFO - Time to execute basesetup : 26.55506205558777
[2022-09-12 02:17:01,343] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:17:01,343] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:17:01,399] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:17:01,399] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:17:01,455] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:17:01,455] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:17:01,511] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 02:17:01,511] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 02:17:01,562] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:01,629] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 02:17:01,629] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 02:17:01,629] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 02:17:06,640] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 02:17:06,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:06,643] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:07,341] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:08,658] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 02:17:08,838] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 02:17:11,676] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 02:17:11,757] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:11,758] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 02:17:11,758] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 02:17:41,781] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 02:17:41,815] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:17:41,841] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:17:41,908] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.720016ms
[2022-09-12 02:17:41,908] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ca95dad5-4e41-49a9-9077-b4dc65f0483f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.720016ms', 'executionTime': '65.663598ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 02:17:41,909] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 02:17:41,934] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 02:17:41,959] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 02:17:42,631] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 670.652797ms
[2022-09-12 02:17:42,631] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 02:17:42,696] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:17:42,730] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:17:42,739] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.460838ms
[2022-09-12 02:17:42,935] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:42,970] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 02:17:42,985] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 02:17:42,986] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 02:17:43,020] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 02:17:43,112] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:43,241] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:43,244] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:43,245] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:44,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:44,091] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:17:44,092] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 02:17:44,218] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:17:44,277] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:44,280] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:44,280] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:45,075] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:45,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 02:17:45,132] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 02:17:45,263] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 02:17:45,321] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 02:17:45,321] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 02:17:45,321] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:17:45,347] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 02:17:45,371] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 02:17:45,381] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.078144ms
[2022-09-12 02:17:45,407] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 02:17:45,431] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 02:17:45,483] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.390057ms
[2022-09-12 02:17:45,552] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 02:17:45,553] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.762004686622463, 'mem_free': 12756107264, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 02:17:45,553] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 02:17:45,557] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:45,557] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:46,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:46,356] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:46,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:47,669] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:47,682] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:47,682] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:48,987] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:48,995] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 02:17:48,995] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 02:17:50,292] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 02:17:57,846] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-12 02:17:57,989] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 02:17:59,203] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 02:17:59,231] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 02:17:59,231] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 02:17:59,290] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:17:59,342] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:17:59,393] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 02:17:59,394] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 02:17:59,474] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 02:17:59,475] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 02:17:59,500] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:17:59,630] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 02:17:59,630] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:17:59,657] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 02:17:59,683] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 02:17:59,683] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:17:59,711] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 02:17:59,737] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 02:17:59,738] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:17:59,764] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 02:17:59,791] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 02:17:59,791] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 02:17:59,817] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 02:17:59,817] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-12 02:17:59,817] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 02:17:59,818] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.164s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 14755 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-12.09.2022-00.32.cfg
Build Log: make-12.09.2022-00.32.log
Server Log: logs-12.09.2022-00.32.tar.gz

Finished