Building

Started building at 2022/09/12 04:04:55
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 12.09.2022-04.04
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at fb7c5c5, changes since last good build: 
fatal: Invalid revision range 1c5995aedb73edda33523300ee73b3da1e3ca5a2..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable 8ee8ab21] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 3f166365] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 46ac95a2] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/12 04:55:36
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 12 05:01:01 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 05:04:55 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 05:05:39 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 12 05:07:12 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 12 05:08:02 b" 'stop_on_failure': 'True'}"
Sep 12 05:08:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 05:08:02 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_1'}"
Sep 12 05:08:02 b'-->result: '
Sep 12 05:08:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 12 05:08:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 05:08:02 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_2'}"
Sep 12 05:08:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:08:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 12 05:08:02 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_3'}"
Sep 12 05:08:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:08:02 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:08:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 12 05:08:02 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_4'}"
Sep 12 05:08:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:08:02 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:08:02 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 05:17:50 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 12 05:18:35 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 12 05:22:19 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 05:26:13 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 05:28:56 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 05:28:56 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_5'}"
Sep 12 05:28:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 12 05:28:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 12 05:28:56 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_6'}"
Sep 12 05:28:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 05:28:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 12 05:28:56 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_7'}"
Sep 12 05:28:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 12 05:28:56 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 12 05:28:56 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_04-55-57/test_8'}"
Sep 12 05:28:56 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 05:28:56 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 05:28:56 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 12 05:28:56 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 12 05:28:56 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 05:28:56 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 05:28:56 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 12 05:28:56 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 12 05:28:56 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 12 05:28:56 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 12 05:28:56 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 12 05:28:56 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 180: go: command not found
/home/buildbot/bin/dotest: line 183: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-12 05:31:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:31:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:31:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:31:49 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 05:31:52 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 05:31:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 05:32:10 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:10 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:11 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:12 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:13 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 05:32:18 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-12 05:32:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 1982001152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:32:19 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 05:32:19 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:32:20 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1801777152, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:32:20 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 05:32:20 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:32:21 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1660239872, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:32:21 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:21 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:32:22 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '20', 'memoryTotal': 15466930176, 'memoryFree': 1602592768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:32:22 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:22 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:32:23 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:32:23 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:32:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:32:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:32:23 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 05:32:23 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 05:32:23 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-12 05:32:23 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 05:32:28 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 05:32:28 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-12 05:32:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:28 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:29 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:32:29 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:32:29 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:32:29 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 05:32:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 05:32:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 05:32:29 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 05:32:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:32:30 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:32:31 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 48.10840769553675, 'mem_free': 13471428608, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 45.40946984291077
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 05:32:34 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 05:32:39 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 05:32:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:32:39 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:32:40 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:32:41 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 05:32:41 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:32:42 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 05:32:43 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:32:43 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 05:32:43 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 164.012868ms
2022-09-12 05:33:13 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'e9a6a322-009c-43cf-9136-0ffb0f1f16b8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '164.012868ms', 'executionTime': '163.869013ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 05:33:13 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 739.767081ms
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.415957ms
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 05:33:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.546897ms
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.583088ms
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.493456081189196, 'mem_free': 13448429568, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:20 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 05:33:20 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 05:33:20 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 05:33:20 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 05:33:20 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 05:33:21 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 05:33:21 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 92.472s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 05:33:21 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.66376755228129, 'mem_free': 13422256128, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:22 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 05:33:27 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 05:33:27 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13422256128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 05:33:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 05:33:29 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13418168320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:33:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13410856960, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13417177088, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:30 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:33:31 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:33:31 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:33:31 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 05:33:31 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-12 05:33:31 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 05:33:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 05:33:36 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 05:33:36 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-12 05:33:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:36 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:37 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 05:33:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 05:33:37 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 05:33:37 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 05:33:38 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 05:33:38 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-12 05:33:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 05:33:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:33:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:33:39 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 05:33:39 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 05:33:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:39 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:40 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.9776824571057, 'mem_free': 13404495872, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 22.45739245414734
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 05:33:44 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 05:33:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 05:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:33:49 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:33:49 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:33:50 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 05:33:50 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 05:33:52 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 05:33:52 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:33:52 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 05:33:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 66.459435ms
2022-09-12 05:34:22 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '19a2c510-7231-4c18-993b-d088e5e51383', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.459435ms', 'executionTime': '66.326339ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 05:34:22 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 686.712223ms
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.947098ms
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 05:34:23 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:34:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee8608c68ee5524ebfbaa13c4945adce67job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-12 05:34:24 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee8608c68ee5524ebfbaa13c4945adce67job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-12 05:34:24 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.177193ms
2022-09-12 05:34:24 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee8608c68ee5524ebfbaa13c4945adce67job_title) USING GSI
2022-09-12 05:34:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee8608c68ee5524ebfbaa13c4945adce67job_title) USING GSI
2022-09-12 05:34:25 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee8608c68ee5524ebfbaa13c4945adce67job_title%29+USING+GSI
2022-09-12 05:34:25 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 24.981908ms
2022-09-12 05:34:26 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee8608c68ee5524ebfbaa13c4945adce67job_title'
2022-09-12 05:34:26 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8608c68ee5524ebfbaa13c4945adce67job_title%27
2022-09-12 05:34:26 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.119172ms
2022-09-12 05:34:27 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee8608c68ee5524ebfbaa13c4945adce67job_title'
2022-09-12 05:34:28 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8608c68ee5524ebfbaa13c4945adce67job_title%27
2022-09-12 05:34:28 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.093905ms
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.686983ms
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '94943c66-dbcd-438f-a3bd-65bcadd623f6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee8608c68ee5524ebfbaa13c4945adce67job_title', 'index_id': '3e8d55a5c4401f2a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.686983ms', 'executionTime': '2.627765ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:29 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-12 05:34:29 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 124.588581ms
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-12 05:34:30 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-12 05:34:31 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:31 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee8608c68ee5524ebfbaa13c4945adce67job_title'
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8608c68ee5524ebfbaa13c4945adce67job_title%27
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.818093ms
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee8608c68ee5524ebfbaa13c4945adce67job_title ON default USING GSI
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee8608c68ee5524ebfbaa13c4945adce67job_title+ON+default+USING+GSI
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 39.907395ms
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee8608c68ee5524ebfbaa13c4945adce67job_title'
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee8608c68ee5524ebfbaa13c4945adce67job_title%27
2022-09-12 05:34:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.514673ms
2022-09-12 05:34:32 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '4b47c094-b0bc-4827-83c1-80957298ce47', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.514673ms', 'executionTime': '5.394832ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:32 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.563443ms
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 52.350289ms
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.19102579820306, 'mem_free': 13282263040, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:33 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 05:34:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 05:34:34 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:34:38 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 05:34:39 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 05:34:39 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 77.877s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_2'}
[2022-09-12 05:34:39,540] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:39,540] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:39,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:39,840] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:34:39,927] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:34:39,927] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-12 05:34:39,928] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:34:40,570] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:34:40,598] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13324754944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:34:40,624] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:34:40,624] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:34:40,625] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:34:40,654] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:34:40,689] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:34:40,690] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:34:40,718] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:34:40,719] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:34:40,719] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:34:40,719] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:34:40,767] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:34:40,770] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:40,770] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:41,048] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:41,049] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:34:41,124] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:34:41,126] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:34:41,158] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:34:41,159] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:34:41,293] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:34:41,321] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13291155456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:34:41,347] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:34:41,374] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:34:41,374] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:34:41,427] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:34:41,430] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:41,431] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:41,724] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:41,725] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:34:41,793] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:34:41,795] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:34:41,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:34:41,826] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:34:41,941] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:34:41,967] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13283295232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:34:41,992] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:34:42,020] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:34:42,022] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:34:42,072] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:34:42,076] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:42,077] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:42,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:42,337] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:34:42,405] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:34:42,406] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:34:42,436] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:34:42,436] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:34:42,550] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:34:42,577] - [task:165] INFO -  {'uptime': '161', 'memoryTotal': 15466930176, 'memoryFree': 13293555712, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:34:42,602] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:34:42,630] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:34:42,631] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:34:42,682] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:34:42,685] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:42,685] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:42,952] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:42,953] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:34:43,024] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:34:43,025] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:34:43,055] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:34:43,056] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:34:43,143] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:34:43,704] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:34:48,709] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:34:48,792] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:34:48,798] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:48,798] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:49,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:49,087] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:34:49,156] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:34:49,157] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:34:49,157] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:34:50,260] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:34:50,317] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:34:50,317] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:34:51,324] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:34:51,654] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:34:52,008] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:34:52,010] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:34:52,064] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:52,065] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:52,315] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:52,320] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:52,320] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:52,573] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:52,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:52,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:52,830] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:52,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:34:52,835] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:34:53,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:34:56,922] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:34:56,923] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.79255381389215, 'mem_free': 13320458240, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:34:56,923] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:34:56,923] - [basetestcase:467] INFO - Time to execute basesetup : 17.385151624679565
[2022-09-12 05:34:56,981] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:34:56,981] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:34:57,039] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:34:57,039] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:34:57,093] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:34:57,093] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:34:57,150] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:34:57,150] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:34:57,202] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:34:57,261] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:34:57,262] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:34:57,262] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:35:02,274] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:35:02,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:02,278] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:02,547] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:03,490] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:35:03,667] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:35:06,287] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:35:06,311] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 05:35:06,966] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:06,967] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:35:06,967] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:35:36,996] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:35:37,024] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:35:37,049] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:35:37,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.700421ms
[2022-09-12 05:35:37,116] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bd79b3e9-db0b-47a8-aab4-702e9e47fe89', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.700421ms', 'executionTime': '64.629676ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:35:37,116] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:35:37,145] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:35:37,175] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:35:37,825] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 646.590622ms
[2022-09-12 05:35:37,825] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:35:37,921] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:35:37,977] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:35:37,988] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.138723ms
[2022-09-12 05:35:38,198] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:38,236] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:35:38,249] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:35:38,250] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:35:38,260] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:35:38,320] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:39,150] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb07545fe927c4f64acdce9ae850b73abjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:35:39,175] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb07545fe927c4f64acdce9ae850b73abjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:35:39,214] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.186969ms
[2022-09-12 05:35:39,215] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb07545fe927c4f64acdce9ae850b73abjob_title) USING GSI
[2022-09-12 05:35:40,245] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb07545fe927c4f64acdce9ae850b73abjob_title) USING GSI
[2022-09-12 05:35:40,269] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb07545fe927c4f64acdce9ae850b73abjob_title%29+USING+GSI
[2022-09-12 05:35:40,299] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.790345ms
[2022-09-12 05:35:41,333] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb07545fe927c4f64acdce9ae850b73abjob_title'
[2022-09-12 05:35:41,360] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb07545fe927c4f64acdce9ae850b73abjob_title%27
[2022-09-12 05:35:41,368] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.866256ms
[2022-09-12 05:35:42,398] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb07545fe927c4f64acdce9ae850b73abjob_title'
[2022-09-12 05:35:42,423] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb07545fe927c4f64acdce9ae850b73abjob_title%27
[2022-09-12 05:35:42,429] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.214891ms
[2022-09-12 05:35:42,882] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 05:35:43,056] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:35:43,979] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:35:44,432] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:44,465] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:35:44,491] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 05:35:44,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.102877ms
[2022-09-12 05:35:44,495] - [task:3246] INFO - {'requestID': 'e41528f9-f2ab-4aa6-b0a9-4ea2ca6709ed', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb07545fe927c4f64acdce9ae850b73abjob_title', 'index_id': '866cfeffb7d03d4a', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.102877ms', 'executionTime': '2.036757ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 05:35:44,495] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:44,495] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:44,496] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:35:44,496] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:35:44,496] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 05:35:44,496] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:35:44,497] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:35:44,497] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:35:44,497] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:35:45,497] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:45,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:35:45,554] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 05:35:45,672] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 108.756474ms
[2022-09-12 05:35:45,672] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:35:45,673] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:35:46,436] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:46,436] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:35:47,467] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb07545fe927c4f64acdce9ae850b73abjob_title'
[2022-09-12 05:35:47,493] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb07545fe927c4f64acdce9ae850b73abjob_title%27
[2022-09-12 05:35:47,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.508351ms
[2022-09-12 05:35:47,529] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb07545fe927c4f64acdce9ae850b73abjob_title ON default USING GSI
[2022-09-12 05:35:47,555] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb07545fe927c4f64acdce9ae850b73abjob_title+ON+default+USING+GSI
[2022-09-12 05:35:47,597] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 37.836161ms
[2022-09-12 05:35:47,633] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb07545fe927c4f64acdce9ae850b73abjob_title'
[2022-09-12 05:35:47,660] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb07545fe927c4f64acdce9ae850b73abjob_title%27
[2022-09-12 05:35:47,667] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.762963ms
[2022-09-12 05:35:47,667] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8ecb270e-0062-4a93-ad5d-145acce4d81f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.762963ms', 'executionTime': '5.640785ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:35:47,772] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:47,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:47,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:48,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:48,177] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:35:48,177] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:35:48,240] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:35:48,296] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:48,299] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:48,299] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:48,644] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:48,704] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:35:48,705] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:35:48,768] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:35:48,824] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:35:48,824] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:35:48,824] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:35:48,851] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:35:48,878] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:35:48,889] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.24443ms
[2022-09-12 05:35:48,916] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:35:48,942] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:35:48,998] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.879548ms
[2022-09-12 05:35:49,065] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:35:49,065] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.06456650391351, 'mem_free': 13200211968, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:35:49,066] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:35:49,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:49,070] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:49,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:49,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:49,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:49,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:49,813] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:49,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:50,215] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:50,221] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:50,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:50,849] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:55,178] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:35:55,324] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:35:55,880] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:35:55,908] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:35:55,908] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:35:55,963] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:35:56,016] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:35:56,071] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:35:56,072] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:35:56,149] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:35:56,150] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:35:56,176] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:56,307] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:35:56,307] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:56,333] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:35:56,359] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:35:56,359] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:56,385] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:35:56,411] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:35:56,411] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:56,442] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:35:56,480] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:35:56,480] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:56,507] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:35:56,507] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:35:56,507] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:35:56,507] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.021s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_3'}
[2022-09-12 05:35:56,606] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:56,606] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:56,914] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:56,944] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:35:57,024] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:35:57,024] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-12 05:35:57,025] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:35:57,635] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:35:57,663] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13200211968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:35:57,688] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:35:57,688] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:35:57,688] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:35:57,717] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:35:57,751] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:35:57,751] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:35:57,780] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:35:57,781] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:35:57,781] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:35:57,781] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:35:57,829] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:35:57,832] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:57,832] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:58,141] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:58,142] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:35:58,217] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:35:58,218] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:35:58,247] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:35:58,248] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:35:58,371] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:35:58,397] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13265641472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:35:58,423] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:35:58,451] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:35:58,451] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:35:58,502] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:35:58,505] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:58,505] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:58,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:58,822] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:35:58,897] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:35:58,899] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:35:58,929] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:35:58,929] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:35:59,046] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:35:59,076] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13265272832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:35:59,101] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:35:59,128] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:35:59,128] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:35:59,179] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:35:59,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:59,185] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:35:59,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:35:59,495] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:35:59,568] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:35:59,569] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:35:59,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:35:59,599] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:35:59,717] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:35:59,743] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13264699392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:35:59,769] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:35:59,797] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:35:59,797] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:35:59,846] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:35:59,849] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:35:59,850] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:00,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:00,181] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:36:00,257] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:36:00,258] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:36:00,291] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:36:00,291] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:36:00,386] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:36:00,937] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:36:05,942] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:36:06,031] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:36:06,039] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:06,039] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:06,362] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:06,363] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:36:06,440] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:36:06,441] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:36:06,441] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:36:07,509] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:36:07,562] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 05:36:07,563] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:36:08,452] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:36:08,764] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:36:09,056] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:36:09,059] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:36:09,118] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:09,118] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:09,492] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:09,497] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:09,497] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:09,829] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:09,834] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:09,834] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:10,265] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:10,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:10,278] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:10,828] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:14,859] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:36:14,859] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.24363095622034, 'mem_free': 13261656064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:36:14,860] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:36:14,860] - [basetestcase:467] INFO - Time to execute basesetup : 18.25634527206421
[2022-09-12 05:36:14,911] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:36:14,912] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:36:14,967] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:36:14,967] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:36:15,022] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:36:15,022] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:36:15,076] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:36:15,076] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:36:15,127] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:36:15,190] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:36:15,190] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:36:15,191] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:36:20,201] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:36:20,204] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:36:20,204] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:36:20,520] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:36:21,463] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:36:21,628] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:36:24,289] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:36:24,305] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 05:36:25,141] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:36:25,142] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:36:25,142] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:36:55,170] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:36:55,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:36:55,225] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:36:55,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.767009ms
[2022-09-12 05:36:55,294] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '568baeb4-9ea0-452a-944e-8f80e967c2e7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.767009ms', 'executionTime': '66.58004ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:36:55,294] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:36:55,320] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:36:55,345] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:36:56,041] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 694.68139ms
[2022-09-12 05:36:56,041] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:36:56,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:36:56,149] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:36:56,157] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.878539ms
[2022-09-12 05:36:56,361] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:36:56,402] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:36:56,432] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:36:56,433] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:36:56,448] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:36:56,509] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:36:57,322] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeeec064c5f47b4819864f35ebd574f54fjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:36:57,348] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeeec064c5f47b4819864f35ebd574f54fjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:36:57,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.315265ms
[2022-09-12 05:36:57,374] - [base_gsi:282] INFO - BUILD INDEX on default(employeeeec064c5f47b4819864f35ebd574f54fjob_title) USING GSI
[2022-09-12 05:36:58,404] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeeec064c5f47b4819864f35ebd574f54fjob_title) USING GSI
[2022-09-12 05:36:58,430] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeeec064c5f47b4819864f35ebd574f54fjob_title%29+USING+GSI
[2022-09-12 05:36:58,453] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.999455ms
[2022-09-12 05:36:59,484] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeec064c5f47b4819864f35ebd574f54fjob_title'
[2022-09-12 05:36:59,509] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeec064c5f47b4819864f35ebd574f54fjob_title%27
[2022-09-12 05:36:59,518] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.495066ms
[2022-09-12 05:37:00,549] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeec064c5f47b4819864f35ebd574f54fjob_title'
[2022-09-12 05:37:00,575] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeec064c5f47b4819864f35ebd574f54fjob_title%27
[2022-09-12 05:37:00,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.571307ms
[2022-09-12 05:37:00,957] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 05:37:01,208] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:37:01,926] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:37:02,586] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:02,620] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:37:02,646] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 05:37:02,651] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.328257ms
[2022-09-12 05:37:02,651] - [task:3246] INFO - {'requestID': '1d33038c-2bbc-4fe5-bb6d-1c1d470d26cf', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeeec064c5f47b4819864f35ebd574f54fjob_title', 'index_id': 'f091f2c8ec803eb7', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.328257ms', 'executionTime': '2.260097ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 05:37:02,651] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:02,651] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:02,652] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:37:02,652] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:37:02,652] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 05:37:02,653] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:37:02,653] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:37:02,653] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:37:02,653] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:37:03,653] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:03,683] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:37:03,709] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 05:37:03,845] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 124.799499ms
[2022-09-12 05:37:03,845] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:37:03,846] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:37:04,640] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:04,641] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:37:05,672] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeec064c5f47b4819864f35ebd574f54fjob_title'
[2022-09-12 05:37:05,698] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeec064c5f47b4819864f35ebd574f54fjob_title%27
[2022-09-12 05:37:05,707] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.28911ms
[2022-09-12 05:37:05,733] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeeec064c5f47b4819864f35ebd574f54fjob_title ON default USING GSI
[2022-09-12 05:37:05,759] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeeec064c5f47b4819864f35ebd574f54fjob_title+ON+default+USING+GSI
[2022-09-12 05:37:05,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.222693ms
[2022-09-12 05:37:05,829] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeeec064c5f47b4819864f35ebd574f54fjob_title'
[2022-09-12 05:37:05,856] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeeec064c5f47b4819864f35ebd574f54fjob_title%27
[2022-09-12 05:37:05,864] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.154018ms
[2022-09-12 05:37:05,864] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '100013f6-211b-431c-b1aa-4eec36f4e1a2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.154018ms', 'executionTime': '6.088648ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:37:05,972] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:37:05,975] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:05,975] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:06,372] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:06,427] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:37:06,428] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:37:06,496] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:37:06,551] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:37:06,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:06,554] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:06,935] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:06,993] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:37:06,993] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:37:07,062] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:37:07,117] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:37:07,118] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:37:07,118] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:37:07,146] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:37:07,182] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:37:07,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.850441ms
[2022-09-12 05:37:07,220] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:37:07,246] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:37:07,292] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.917562ms
[2022-09-12 05:37:07,359] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:37:07,359] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.63252088324522, 'mem_free': 13134024704, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:37:07,359] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:37:07,364] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:07,364] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:07,791] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:07,797] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:07,797] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:08,397] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:08,405] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:08,405] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:09,118] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:09,125] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:09,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:09,835] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:13,964] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:37:14,111] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:37:15,112] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:37:15,141] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:37:15,141] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:37:15,201] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:37:15,271] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:37:15,327] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:37:15,328] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:37:15,410] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:37:15,411] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:37:15,437] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:15,575] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:37:15,575] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:15,603] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:37:15,628] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:37:15,628] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:15,655] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:37:15,681] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:37:15,681] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:15,708] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:37:15,737] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:37:15,737] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:15,764] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:37:15,765] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:37:15,765] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:37:15,765] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.215s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_4'}
[2022-09-12 05:37:15,865] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:15,865] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:16,239] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:16,270] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:37:16,349] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:37:16,349] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-12 05:37:16,350] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:37:16,894] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:37:16,921] - [task:165] INFO -  {'uptime': '323', 'memoryTotal': 15466930176, 'memoryFree': 13113643008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:37:16,946] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:37:16,946] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:37:16,947] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:37:16,977] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:37:17,019] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:37:17,020] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:37:17,047] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:37:17,048] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:37:17,049] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:37:17,049] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:37:17,097] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:37:17,099] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:17,099] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:17,464] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:17,465] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:37:17,547] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:37:17,548] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:37:17,576] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:37:17,577] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:37:17,702] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:37:17,729] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13216428032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:37:17,755] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:37:17,783] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:37:17,783] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:37:17,834] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:37:17,837] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:17,838] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:18,194] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:18,195] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:37:18,276] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:37:18,277] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:37:18,307] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:37:18,308] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:37:18,423] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:37:18,451] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13215965184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:37:18,480] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:37:18,508] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:37:18,508] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:37:18,559] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:37:18,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:18,563] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:18,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:18,919] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:37:19,000] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:37:19,001] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:37:19,030] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:37:19,031] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:37:19,152] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:37:19,182] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13215604736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:37:19,210] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:37:19,240] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:37:19,241] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:37:19,293] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:37:19,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:19,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:19,705] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:19,706] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:37:19,797] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:37:19,798] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:37:19,829] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:37:19,830] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:37:19,924] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:37:20,482] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:37:25,487] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:37:25,571] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:37:25,577] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:25,577] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:25,952] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:25,953] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:37:26,036] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:37:26,037] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:37:26,037] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:37:27,071] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:37:27,130] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:37:27,131] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:37:28,051] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:37:28,366] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:37:28,651] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:37:28,654] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:37:28,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:28,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:29,102] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:29,107] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:29,107] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:29,510] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:29,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:29,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:30,120] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:30,130] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:30,130] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:30,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:35,291] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:37:35,291] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.01499993551446, 'mem_free': 13217808384, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:37:35,291] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:37:35,292] - [basetestcase:467] INFO - Time to execute basesetup : 19.428767919540405
[2022-09-12 05:37:35,343] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:37:35,343] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:37:35,395] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:37:35,396] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:37:35,449] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:37:35,449] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:37:35,504] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:37:35,504] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:37:35,556] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:37:35,615] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:37:35,615] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:37:35,616] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:37:40,627] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:37:40,631] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:37:40,631] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:37:41,019] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:37:41,988] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:37:42,231] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:37:44,770] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:37:44,790] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 05:37:45,469] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:37:45,469] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:37:45,469] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:38:15,500] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:38:15,528] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:38:15,553] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:38:15,620] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.684212ms
[2022-09-12 05:38:15,621] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cabce349-4370-4e23-b66c-a6a0f1d3e296', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.684212ms', 'executionTime': '65.613686ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:38:15,621] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:38:15,647] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:38:15,672] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:38:16,348] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 673.935516ms
[2022-09-12 05:38:16,348] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:38:16,403] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:38:16,439] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:38:16,451] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.741738ms
[2022-09-12 05:38:16,654] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:16,687] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:38:16,702] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:38:16,702] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:38:16,717] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:38:16,794] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:17,619] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee836d711809de435e9f2be8d7c004d433job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:38:17,644] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee836d711809de435e9f2be8d7c004d433job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:38:17,674] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.022853ms
[2022-09-12 05:38:17,675] - [base_gsi:282] INFO - BUILD INDEX on default(employee836d711809de435e9f2be8d7c004d433job_title) USING GSI
[2022-09-12 05:38:18,705] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee836d711809de435e9f2be8d7c004d433job_title) USING GSI
[2022-09-12 05:38:18,731] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee836d711809de435e9f2be8d7c004d433job_title%29+USING+GSI
[2022-09-12 05:38:18,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.558283ms
[2022-09-12 05:38:19,785] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee836d711809de435e9f2be8d7c004d433job_title'
[2022-09-12 05:38:19,810] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee836d711809de435e9f2be8d7c004d433job_title%27
[2022-09-12 05:38:19,819] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.586109ms
[2022-09-12 05:38:20,851] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee836d711809de435e9f2be8d7c004d433job_title'
[2022-09-12 05:38:20,878] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee836d711809de435e9f2be8d7c004d433job_title%27
[2022-09-12 05:38:20,886] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.774293ms
[2022-09-12 05:38:21,354] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 05:38:21,526] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:38:22,410] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:38:22,602] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-12 05:38:22,602] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:38:22,781] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-12 05:38:22,782] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-12 05:38:22,782] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2093674396, 0, b'')
[2022-09-12 05:38:22,888] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:22,916] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:38:22,941] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 05:38:22,946] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.653668ms
[2022-09-12 05:38:22,946] - [task:3246] INFO - {'requestID': '6992bf22-9407-40a6-8080-5af59d3952bc', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee836d711809de435e9f2be8d7c004d433job_title', 'index_id': 'fbe3332bef5de6ea', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.653668ms', 'executionTime': '2.584529ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 05:38:22,946] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:22,946] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:22,947] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:38:22,947] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:38:22,947] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 05:38:22,948] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:38:22,948] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:38:22,948] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:38:22,948] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:38:23,948] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:23,980] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:38:24,006] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 05:38:24,118] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 102.886732ms
[2022-09-12 05:38:24,119] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:38:24,119] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:38:24,898] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:24,899] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:38:25,930] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee836d711809de435e9f2be8d7c004d433job_title'
[2022-09-12 05:38:25,956] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee836d711809de435e9f2be8d7c004d433job_title%27
[2022-09-12 05:38:25,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.769541ms
[2022-09-12 05:38:25,995] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee836d711809de435e9f2be8d7c004d433job_title ON default USING GSI
[2022-09-12 05:38:26,021] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee836d711809de435e9f2be8d7c004d433job_title+ON+default+USING+GSI
[2022-09-12 05:38:26,059] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.774614ms
[2022-09-12 05:38:26,098] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee836d711809de435e9f2be8d7c004d433job_title'
[2022-09-12 05:38:26,125] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee836d711809de435e9f2be8d7c004d433job_title%27
[2022-09-12 05:38:26,133] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.028086ms
[2022-09-12 05:38:26,133] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6c530057-3d81-4aef-8306-7f5f8b5cd0ce', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.028086ms', 'executionTime': '5.964611ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:38:26,244] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:26,248] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:26,248] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:26,740] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:26,801] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:38:26,801] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:38:26,880] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:38:26,935] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:26,938] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:26,939] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:27,421] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:27,483] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:38:27,483] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:38:27,563] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:38:27,618] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:27,619] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:38:27,619] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:38:27,644] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:38:27,670] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:38:27,679] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.169039ms
[2022-09-12 05:38:27,706] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:38:27,731] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:38:27,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.095455ms
[2022-09-12 05:38:27,856] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:38:27,856] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.06230515700097, 'mem_free': 13087981568, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:38:27,857] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:38:27,861] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:27,861] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:28,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:28,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:28,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:28,952] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:28,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:28,960] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:29,737] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:29,745] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:29,745] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:30,505] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:35,024] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:38:35,165] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:38:35,687] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:38:35,716] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:38:35,716] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:38:35,772] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:38:35,828] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:38:35,881] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:38:35,882] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:38:35,966] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:38:35,967] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:38:35,996] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,129] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:38:36,129] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,155] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:38:36,181] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:38:36,181] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,209] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:38:36,237] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:38:36,237] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,265] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:38:36,302] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:38:36,302] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,332] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:38:36,332] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:38:36,332] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:38:36,333] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.525s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_5'}
[2022-09-12 05:38:36,436] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:36,437] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:36,838] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:36,868] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:38:36,950] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:38:36,950] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-12 05:38:36,951] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:38:37,463] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:38:37,489] - [task:165] INFO -  {'uptime': '404', 'memoryTotal': 15466930176, 'memoryFree': 13075865600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:38:37,514] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:38:37,515] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:38:37,515] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:38:37,551] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:38:37,585] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:38:37,585] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:38:37,613] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:38:37,614] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:38:37,614] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:38:37,614] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:38:37,662] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:38:37,665] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:37,665] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:38,065] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:38,066] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:38:38,159] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:38:38,160] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:38:38,190] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:38:38,191] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:38:38,312] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:38:38,338] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13188808704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:38:38,363] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:38:38,393] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:38:38,393] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:38:38,442] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:38:38,445] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:38,445] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:38,877] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:38,878] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:38:38,976] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:38:38,978] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:38:39,009] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:38:39,009] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:38:39,127] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:38:39,155] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13189185536, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:38:39,183] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:38:39,211] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:38:39,211] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:38:39,262] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:38:39,267] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:39,267] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:39,695] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:39,697] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:38:39,784] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:38:39,785] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:38:39,814] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:38:39,815] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:38:39,935] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:38:39,963] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13189578752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:38:39,988] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:38:40,021] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:38:40,022] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:38:40,075] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:38:40,078] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:40,078] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:40,490] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:40,491] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:38:40,580] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:38:40,581] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:38:40,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:38:40,612] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:38:40,701] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:38:41,223] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:38:46,225] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:38:46,308] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:38:46,313] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:46,313] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:46,737] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:46,738] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:38:46,831] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:38:46,832] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:38:46,832] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:38:47,841] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:38:47,895] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 05:38:47,896] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:38:48,795] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:38:49,109] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:38:49,388] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:38:49,391] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:38:49,456] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:49,456] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:49,906] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:49,911] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:49,911] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:50,375] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:50,380] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:50,381] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:51,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:51,147] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:38:51,147] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:38:51,945] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:38:56,674] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:38:56,674] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.00764611370717, 'mem_free': 13192970240, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:38:56,675] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:38:56,675] - [basetestcase:467] INFO - Time to execute basesetup : 20.240952253341675
[2022-09-12 05:38:56,725] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:38:56,725] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:38:56,777] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:38:56,777] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:38:56,829] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:38:56,829] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:38:56,882] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:38:56,883] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:38:56,933] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:38:56,992] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:38:56,996] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:38:56,996] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:39:02,003] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:39:02,007] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:02,007] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:02,447] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:03,449] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:39:03,705] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:39:06,145] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:39:06,162] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 05:39:07,225] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:07,225] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:39:07,225] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:39:37,255] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:39:37,285] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:39:37,312] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:39:37,379] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.268384ms
[2022-09-12 05:39:37,380] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2448aadd-ae10-4e6c-ac08-f594409ee406', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.268384ms', 'executionTime': '65.199678ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:39:37,380] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:39:37,406] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:39:37,433] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:39:38,143] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 708.289591ms
[2022-09-12 05:39:38,143] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:39:38,203] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:39:38,237] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:39:38,244] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.661783ms
[2022-09-12 05:39:38,435] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:38,468] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:39:38,482] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:39:38,482] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:39:38,499] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:39:38,574] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:39,399] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee14084794958d472e9be3a65215a2e021job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:39:39,426] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee14084794958d472e9be3a65215a2e021job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:39:39,462] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.075725ms
[2022-09-12 05:39:39,463] - [base_gsi:282] INFO - BUILD INDEX on default(employee14084794958d472e9be3a65215a2e021job_title) USING GSI
[2022-09-12 05:39:40,493] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee14084794958d472e9be3a65215a2e021job_title) USING GSI
[2022-09-12 05:39:40,519] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee14084794958d472e9be3a65215a2e021job_title%29+USING+GSI
[2022-09-12 05:39:40,541] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.152576ms
[2022-09-12 05:39:41,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee14084794958d472e9be3a65215a2e021job_title'
[2022-09-12 05:39:41,597] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee14084794958d472e9be3a65215a2e021job_title%27
[2022-09-12 05:39:41,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.59616ms
[2022-09-12 05:39:42,634] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee14084794958d472e9be3a65215a2e021job_title'
[2022-09-12 05:39:42,659] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee14084794958d472e9be3a65215a2e021job_title%27
[2022-09-12 05:39:42,666] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.427104ms
[2022-09-12 05:39:43,159] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:39:43,323] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:39:46,020] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:39:46,671] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:46,703] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:39:46,729] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 05:39:46,733] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.800003ms
[2022-09-12 05:39:46,733] - [task:3246] INFO - {'requestID': '634a0d50-b91c-4af2-99ea-84afe5c33b3a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee14084794958d472e9be3a65215a2e021job_title', 'index_id': 'a26ff97324fe8fa2', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.800003ms', 'executionTime': '2.73854ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 05:39:46,733] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:46,734] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:46,734] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:39:46,735] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:39:46,735] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 05:39:46,735] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:39:46,735] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:39:46,736] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:39:46,736] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:39:47,735] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:47,765] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:39:47,791] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 05:39:47,969] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 163.081745ms
[2022-09-12 05:39:47,969] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:39:47,970] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:39:49,500] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:49,501] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:39:50,530] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee14084794958d472e9be3a65215a2e021job_title'
[2022-09-12 05:39:50,556] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee14084794958d472e9be3a65215a2e021job_title%27
[2022-09-12 05:39:50,565] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.205075ms
[2022-09-12 05:39:50,591] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee14084794958d472e9be3a65215a2e021job_title ON default USING GSI
[2022-09-12 05:39:50,616] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee14084794958d472e9be3a65215a2e021job_title+ON+default+USING+GSI
[2022-09-12 05:39:50,665] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.956482ms
[2022-09-12 05:39:50,697] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee14084794958d472e9be3a65215a2e021job_title'
[2022-09-12 05:39:50,723] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee14084794958d472e9be3a65215a2e021job_title%27
[2022-09-12 05:39:50,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.758492ms
[2022-09-12 05:39:50,730] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '837f3241-a800-4437-a9bb-f07cdf160e7c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.758492ms', 'executionTime': '5.704868ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:39:50,831] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:50,835] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:50,835] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:51,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:51,451] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:39:51,451] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:39:51,541] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:39:51,598] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:51,601] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:51,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:52,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:52,179] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:39:52,179] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:39:52,267] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:39:52,321] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:39:52,322] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:39:52,322] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:39:52,347] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:39:52,372] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:39:52,382] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.967292ms
[2022-09-12 05:39:52,407] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:39:52,431] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:39:52,487] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.060785ms
[2022-09-12 05:39:52,550] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:39:52,551] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 38.26509499689645, 'mem_free': 13020303360, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:39:52,551] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:39:52,555] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:52,555] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:53,098] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:53,103] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:53,103] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:53,599] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:53,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:53,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:54,298] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:39:54,306] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:39:54,306] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:39:55,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:01,437] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:40:01,580] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:40:02,436] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:40:02,465] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:40:02,465] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:40:02,524] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:40:02,580] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:40:02,638] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:40:02,639] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:40:02,734] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:40:02,735] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:40:02,765] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:02,909] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:40:02,909] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:02,937] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:40:02,963] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:40:02,963] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:02,989] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:40:03,017] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:40:03,018] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:03,044] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:40:03,070] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:40:03,070] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:03,097] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:40:03,097] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:40:03,097] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:40:03,098] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.716s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_6'}
[2022-09-12 05:40:03,202] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:03,202] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:03,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:03,722] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:40:03,800] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:40:03,800] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-12 05:40:03,801] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:40:04,228] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:40:04,255] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13126672384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:40:04,280] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:40:04,280] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:40:04,281] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:40:04,318] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:40:04,352] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:40:04,352] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:40:04,380] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:40:04,381] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:40:04,381] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:40:04,381] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:40:04,429] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:40:04,432] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:04,432] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:04,937] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:04,938] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:40:05,038] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:40:05,039] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:40:05,068] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:40:05,068] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:40:05,189] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:40:05,215] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13024829440, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:40:05,240] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:40:05,269] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:40:05,270] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:40:05,320] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:40:05,323] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:05,323] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:05,833] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:05,834] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:40:05,936] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:40:05,937] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:40:05,966] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:40:05,967] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:40:06,086] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:40:06,112] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13120774144, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:40:06,138] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:40:06,164] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:40:06,165] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:40:06,218] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:40:06,222] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:06,222] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:06,778] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:06,779] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:40:06,889] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:40:06,890] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:40:06,919] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:40:06,920] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:40:07,039] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:40:07,067] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13130735616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:40:07,094] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:40:07,123] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:40:07,123] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:40:07,177] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:40:07,180] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:07,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:07,701] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:07,702] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:40:07,815] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:40:07,816] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:40:07,847] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:40:07,848] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:40:07,948] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:40:08,622] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:40:13,627] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:40:13,709] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:40:13,714] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:13,714] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:14,231] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:14,232] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:40:14,338] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:40:14,339] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:40:14,339] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:40:15,062] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:40:15,117] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 05:40:15,118] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:40:16,031] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:40:16,353] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:40:16,622] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:40:16,624] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:40:16,679] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:16,679] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:17,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:17,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:17,185] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:17,829] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:17,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:17,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:18,733] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:18,741] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:18,741] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:19,605] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:25,115] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:40:25,116] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.36137063301565, 'mem_free': 13127020544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:40:25,116] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:40:25,116] - [basetestcase:467] INFO - Time to execute basesetup : 21.917291402816772
[2022-09-12 05:40:25,170] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:40:25,171] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:40:25,227] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:40:25,227] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:40:25,287] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:40:25,287] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:40:25,343] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:40:25,343] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:40:25,394] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:40:25,455] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:40:25,455] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:40:25,456] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:40:30,463] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:40:30,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:40:30,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:40:30,983] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:40:31,972] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:40:32,234] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:40:34,559] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:40:34,579] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 05:40:35,974] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:40:35,974] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:40:35,974] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:41:05,996] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:41:06,026] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:41:06,052] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:41:06,123] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.653363ms
[2022-09-12 05:41:06,124] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '65365429-709a-4118-afe9-3e086834036c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.653363ms', 'executionTime': '69.586776ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:41:06,124] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:41:06,151] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:41:06,178] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:41:06,882] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 701.945208ms
[2022-09-12 05:41:06,883] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:41:06,953] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:41:07,003] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:41:07,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 10.001319ms
[2022-09-12 05:41:07,207] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:41:07,240] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:41:07,260] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:41:07,260] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:41:07,275] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:41:07,354] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:41:08,169] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec410c8ceb21946db9d1c7ada8ab87bc7job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:41:08,196] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:41:08,231] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.105188ms
[2022-09-12 05:41:08,231] - [base_gsi:282] INFO - BUILD INDEX on default(employeec410c8ceb21946db9d1c7ada8ab87bc7job_title) USING GSI
[2022-09-12 05:41:09,262] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec410c8ceb21946db9d1c7ada8ab87bc7job_title) USING GSI
[2022-09-12 05:41:09,289] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%29+USING+GSI
[2022-09-12 05:41:09,312] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.000028ms
[2022-09-12 05:41:10,342] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec410c8ceb21946db9d1c7ada8ab87bc7job_title'
[2022-09-12 05:41:10,368] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%27
[2022-09-12 05:41:10,378] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.521499ms
[2022-09-12 05:41:11,411] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec410c8ceb21946db9d1c7ada8ab87bc7job_title'
[2022-09-12 05:41:11,436] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%27
[2022-09-12 05:41:11,443] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.351407ms
[2022-09-12 05:41:11,938] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 05:41:12,103] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:41:13,231] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:41:13,853] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 05:41:14,034] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:41:15,336] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:41:15,748] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:41:15,924] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:41:19,544] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:41:20,462] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:20,495] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:41:20,521] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 05:41:20,525] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.971457ms
[2022-09-12 05:41:20,526] - [task:3246] INFO - {'requestID': '0414d357-863e-4ab1-ac88-0ee46524d552', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec410c8ceb21946db9d1c7ada8ab87bc7job_title', 'index_id': '3bc5ece4bcb36cfb', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.971457ms', 'executionTime': '2.903583ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 05:41:20,526] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:20,526] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:20,527] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:41:20,527] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:41:20,528] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 05:41:20,528] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:41:20,528] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:41:20,528] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:41:20,528] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 05:41:21,527] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:21,557] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 05:41:21,582] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 05:41:21,938] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 133.402776ms
[2022-09-12 05:41:21,938] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:41:21,939] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:41:23,514] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:23,514] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 05:41:24,544] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec410c8ceb21946db9d1c7ada8ab87bc7job_title'
[2022-09-12 05:41:24,570] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%27
[2022-09-12 05:41:24,579] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.216249ms
[2022-09-12 05:41:24,604] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec410c8ceb21946db9d1c7ada8ab87bc7job_title ON default USING GSI
[2022-09-12 05:41:24,631] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec410c8ceb21946db9d1c7ada8ab87bc7job_title+ON+default+USING+GSI
[2022-09-12 05:41:24,678] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.511424ms
[2022-09-12 05:41:24,716] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec410c8ceb21946db9d1c7ada8ab87bc7job_title'
[2022-09-12 05:41:24,741] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec410c8ceb21946db9d1c7ada8ab87bc7job_title%27
[2022-09-12 05:41:24,749] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.79939ms
[2022-09-12 05:41:24,749] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '02018e5a-8f51-485d-a395-be96b6843a00', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.79939ms', 'executionTime': '5.731731ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:41:24,848] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:41:24,850] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:24,850] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:25,488] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:25,542] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:41:25,542] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:41:25,646] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:41:25,707] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:41:25,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:25,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:26,352] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:26,409] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:41:26,409] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:41:26,516] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:41:26,571] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:41:26,572] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:41:26,572] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:41:26,597] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:41:26,622] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:41:26,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.735097ms
[2022-09-12 05:41:26,656] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:41:26,681] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:41:26,734] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.349561ms
[2022-09-12 05:41:26,803] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:41:26,803] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.84845836503528, 'mem_free': 12861517824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:41:26,803] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:41:26,807] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:26,807] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:27,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:27,479] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:27,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:28,145] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:28,149] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:28,150] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:29,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:29,048] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:29,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:30,156] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:37,654] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:41:37,794] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:41:38,777] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:41:38,805] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:41:38,805] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:41:38,859] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:41:38,918] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:41:38,973] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:41:38,974] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:41:39,052] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:41:39,053] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:41:39,078] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:39,208] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:41:39,208] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:39,234] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:41:39,259] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:41:39,259] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:39,285] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:41:39,311] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:41:39,311] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:39,336] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:41:39,362] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:41:39,362] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:39,388] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:41:39,389] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 05:41:39,389] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:41:39,389] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 96.241s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_7'}
[2022-09-12 05:41:39,520] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:39,520] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:40,117] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:40,150] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:41:40,233] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:41:40,234] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-12 05:41:40,234] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:41:40,546] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:41:40,572] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 13068271616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:41:40,597] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:41:40,597] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:41:40,598] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:41:40,631] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:41:40,667] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:41:40,667] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:41:40,695] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:41:40,697] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:41:40,697] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:41:40,697] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:41:40,745] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:41:40,748] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:40,748] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:41,393] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:41,394] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:41:41,517] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:41:41,518] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:41:41,548] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:41:41,549] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:41:41,668] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:41:41,695] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 13068066816, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:41:41,721] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:41:41,748] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:41:41,748] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:41:41,799] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:41:41,802] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:41,802] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:42,404] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:42,405] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:41:42,529] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:41:42,530] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:41:42,562] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:41:42,562] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:41:42,683] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:41:42,710] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 13067948032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:41:42,738] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:41:42,770] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:41:42,770] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:41:42,825] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:41:42,829] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:42,829] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:43,466] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:43,467] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:41:43,579] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:41:43,580] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:41:43,611] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:41:43,612] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:41:43,728] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:41:43,756] - [task:165] INFO -  {'uptime': '583', 'memoryTotal': 15466930176, 'memoryFree': 13068709888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:41:43,783] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:41:43,814] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:41:43,814] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:41:43,864] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:41:43,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:43,867] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:44,448] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:44,449] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:41:44,561] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:41:44,562] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:41:44,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:41:44,593] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:41:44,680] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:41:45,213] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:41:50,218] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:41:50,305] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:41:50,312] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:50,312] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:50,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:50,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:41:51,030] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:41:51,031] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:41:51,032] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:41:51,833] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:41:51,888] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 05:41:51,889] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:41:52,804] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:41:53,118] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:41:53,401] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:41:53,405] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 05:41:53,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:53,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:54,120] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:54,125] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:54,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:55,002] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:55,015] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:55,015] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:56,116] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:41:56,130] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:41:56,130] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:41:57,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:03,826] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:42:03,827] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.88775402037463, 'mem_free': 13081919488, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:42:03,827] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:42:03,827] - [basetestcase:467] INFO - Time to execute basesetup : 24.309410572052002
[2022-09-12 05:42:03,878] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:42:03,878] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:42:03,933] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:42:03,933] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:42:03,988] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:42:03,988] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:42:04,040] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:42:04,041] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:42:04,091] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:04,155] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:42:04,155] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:42:04,156] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:42:09,167] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:42:09,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:09,171] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:09,812] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:10,852] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:42:11,023] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:42:13,219] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:42:13,305] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:13,305] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:42:13,305] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:42:43,330] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:42:43,361] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:42:43,386] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:42:43,453] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.455305ms
[2022-09-12 05:42:43,453] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f2207e77-baeb-42c6-9d62-00081a8ed7cb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.455305ms', 'executionTime': '65.396746ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:42:43,454] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:42:43,479] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:42:43,504] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:42:44,145] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 638.925867ms
[2022-09-12 05:42:44,145] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:42:44,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:42:44,265] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:42:44,273] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.185147ms
[2022-09-12 05:42:44,469] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:44,508] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:42:44,527] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:42:44,529] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:42:44,544] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:42:44,548] - [base_gsi:326] INFO - []
[2022-09-12 05:42:45,431] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4c218715f4dd42bd8fae3f836ebd8729job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:42:45,460] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4c218715f4dd42bd8fae3f836ebd8729job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:42:45,490] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.314019ms
[2022-09-12 05:42:45,524] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4c218715f4dd42bd8fae3f836ebd8729join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:42:45,550] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4c218715f4dd42bd8fae3f836ebd8729join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:42:45,597] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.893181ms
[2022-09-12 05:42:45,598] - [base_gsi:282] INFO - BUILD INDEX on default(employee4c218715f4dd42bd8fae3f836ebd8729job_title,employee4c218715f4dd42bd8fae3f836ebd8729join_yr) USING GSI
[2022-09-12 05:42:46,629] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee4c218715f4dd42bd8fae3f836ebd8729job_title,employee4c218715f4dd42bd8fae3f836ebd8729join_yr) USING GSI
[2022-09-12 05:42:46,654] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee4c218715f4dd42bd8fae3f836ebd8729job_title%2Cemployee4c218715f4dd42bd8fae3f836ebd8729join_yr%29+USING+GSI
[2022-09-12 05:42:46,693] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.946497ms
[2022-09-12 05:42:47,724] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729job_title'
[2022-09-12 05:42:47,749] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729job_title%27
[2022-09-12 05:42:47,758] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.736385ms
[2022-09-12 05:42:48,792] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729job_title'
[2022-09-12 05:42:48,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729job_title%27
[2022-09-12 05:42:48,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.121793ms
[2022-09-12 05:42:48,857] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729join_yr'
[2022-09-12 05:42:48,883] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729join_yr%27
[2022-09-12 05:42:48,886] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.802327ms
[2022-09-12 05:42:49,918] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729job_title'
[2022-09-12 05:42:49,945] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729job_title%27
[2022-09-12 05:42:49,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.375642ms
[2022-09-12 05:42:49,982] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4c218715f4dd42bd8fae3f836ebd8729job_title ON default USING GSI
[2022-09-12 05:42:50,011] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4c218715f4dd42bd8fae3f836ebd8729job_title+ON+default+USING+GSI
[2022-09-12 05:42:50,058] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.312886ms
[2022-09-12 05:42:50,092] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729join_yr'
[2022-09-12 05:42:50,118] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729join_yr%27
[2022-09-12 05:42:50,126] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.679089ms
[2022-09-12 05:42:50,152] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee4c218715f4dd42bd8fae3f836ebd8729join_yr ON default USING GSI
[2022-09-12 05:42:50,178] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee4c218715f4dd42bd8fae3f836ebd8729join_yr+ON+default+USING+GSI
[2022-09-12 05:42:50,222] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.871182ms
[2022-09-12 05:42:50,257] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729job_title'
[2022-09-12 05:42:50,282] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729job_title%27
[2022-09-12 05:42:50,290] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.717654ms
[2022-09-12 05:42:50,290] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1e45387c-e19e-4a3d-aa88-870fc1036df5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.717654ms', 'executionTime': '5.662596ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:42:50,316] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4c218715f4dd42bd8fae3f836ebd8729join_yr'
[2022-09-12 05:42:50,341] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4c218715f4dd42bd8fae3f836ebd8729join_yr%27
[2022-09-12 05:42:50,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.217553ms
[2022-09-12 05:42:50,344] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f2ffe933-b679-43bf-b648-c71380324383', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.217553ms', 'executionTime': '1.165874ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:42:50,444] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:50,447] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:50,448] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:51,071] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:51,132] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:42:51,132] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:42:51,238] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:42:51,301] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:51,304] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:51,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:51,963] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:52,021] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:42:52,022] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:42:52,129] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:42:52,188] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:42:52,188] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:42:52,188] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:42:52,215] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:42:52,241] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:42:52,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.844362ms
[2022-09-12 05:42:52,282] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:42:52,308] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:42:52,353] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.791931ms
[2022-09-12 05:42:52,430] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:42:52,430] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.30577322026378, 'mem_free': 12956864512, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:42:52,430] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:42:52,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:52,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:53,057] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:53,063] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:53,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:53,663] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:53,669] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:53,669] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:54,552] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:42:54,564] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:42:54,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:42:55,652] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:03,015] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-12 05:43:03,164] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:43:03,559] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:43:03,587] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:43:03,587] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:43:03,646] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:43:03,700] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:43:03,753] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:43:03,754] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:43:03,832] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:43:03,834] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:43:03,859] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:03,993] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:43:03,993] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:04,020] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:43:04,045] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:43:04,045] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:04,071] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:43:04,096] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:43:04,097] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:04,125] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:43:04,151] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:43:04,151] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:04,178] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:43:04,179] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 05:43:04,179] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:43:04,179] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.713s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_8'}
[2022-09-12 05:43:04,259] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:04,259] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:04,854] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:04,888] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:43:04,966] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:43:04,967] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-12 05:43:04,967] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:43:05,286] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:43:05,314] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 13087723520, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:43:05,340] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:43:05,340] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:43:05,340] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:43:05,373] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:43:05,408] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:43:05,409] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:43:05,437] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:43:05,438] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:43:05,438] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:43:05,438] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:43:05,486] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:43:05,489] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:05,489] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:06,131] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:06,132] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:43:06,261] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:43:06,263] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:43:06,294] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:43:06,295] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:43:06,418] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:43:06,446] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 13088051200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:43:06,474] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:43:06,502] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:43:06,502] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:43:06,555] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:43:06,558] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:06,558] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:07,213] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:07,214] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:43:07,341] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:43:07,343] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:43:07,374] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:43:07,375] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:43:07,495] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:43:07,522] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13088563200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:43:07,548] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:43:07,576] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:43:07,576] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:43:07,628] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:43:07,633] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:07,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:08,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:08,257] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:43:08,387] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:43:08,388] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:43:08,420] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:43:08,421] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:43:08,538] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:43:08,569] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13079535616, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:43:08,599] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:43:08,632] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:43:08,632] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:43:08,684] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:43:08,688] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:08,688] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:09,317] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:09,318] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:43:09,442] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:43:09,443] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:43:09,473] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:43:09,473] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:43:09,566] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:43:10,102] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:43:15,107] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:43:15,194] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:43:15,199] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:15,200] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:15,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:15,817] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:43:15,937] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:43:15,938] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:43:15,938] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:43:16,727] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:43:16,780] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 05:43:16,780] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:43:17,825] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:43:18,133] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:43:18,429] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:43:18,433] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 05:43:18,504] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:18,504] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:19,112] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:19,117] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:19,117] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:19,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:19,994] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:19,994] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:21,032] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:21,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:21,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:22,082] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:28,522] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:43:28,522] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.411387306436, 'mem_free': 13085700096, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:43:28,523] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:43:28,523] - [basetestcase:467] INFO - Time to execute basesetup : 24.2667396068573
[2022-09-12 05:43:28,575] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:43:28,575] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:43:28,627] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:43:28,627] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:43:28,680] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:43:28,680] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:43:28,739] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:43:28,739] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:43:28,792] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:43:28,856] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:43:28,856] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:43:28,857] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:43:33,868] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:43:33,872] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:43:33,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:43:34,523] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:43:35,543] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:43:35,822] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:43:37,977] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:43:38,052] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:43:38,053] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:43:38,053] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:44:08,083] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:44:08,113] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:44:08,138] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:44:08,204] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.856579ms
[2022-09-12 05:44:08,204] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '43baeefb-2d40-44e8-990a-47fabe3fda01', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.856579ms', 'executionTime': '63.784668ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:44:08,205] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:44:08,230] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:44:08,254] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:44:08,903] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 646.697279ms
[2022-09-12 05:44:08,903] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:44:08,992] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:44:09,028] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:44:09,037] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.344505ms
[2022-09-12 05:44:09,257] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:44:09,293] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:44:09,311] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:44:09,311] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:44:09,336] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:44:09,337] - [base_gsi:326] INFO - []
[2022-09-12 05:44:10,210] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:44:10,237] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:44:10,271] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.175526ms
[2022-09-12 05:44:10,271] - [base_gsi:282] INFO - BUILD INDEX on default(employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr) USING GSI
[2022-09-12 05:44:11,301] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr) USING GSI
[2022-09-12 05:44:11,328] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%29+USING+GSI
[2022-09-12 05:44:11,352] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.510923ms
[2022-09-12 05:44:12,384] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr'
[2022-09-12 05:44:12,410] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%27
[2022-09-12 05:44:12,419] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.476931ms
[2022-09-12 05:44:13,449] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr'
[2022-09-12 05:44:13,475] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%27
[2022-09-12 05:44:13,484] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.015417ms
[2022-09-12 05:44:14,515] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr'
[2022-09-12 05:44:14,541] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%27
[2022-09-12 05:44:14,551] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.82016ms
[2022-09-12 05:44:14,576] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr ON default USING GSI
[2022-09-12 05:44:14,602] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr+ON+default+USING+GSI
[2022-09-12 05:44:14,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 34.554701ms
[2022-09-12 05:44:14,675] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr'
[2022-09-12 05:44:14,700] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee1a585cac8d0c4b68b3593392c901f46ejob_title_join_yr%27
[2022-09-12 05:44:14,708] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.368165ms
[2022-09-12 05:44:14,709] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3f85c825-63d8-484b-97ed-2c2b6c32605d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.368165ms', 'executionTime': '6.300022ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:44:14,810] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:44:14,813] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:14,813] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:15,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:15,480] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:44:15,481] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:44:15,588] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:44:15,645] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:44:15,648] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:15,649] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:16,329] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:16,385] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:44:16,386] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:44:16,501] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:44:16,557] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:44:16,558] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:44:16,558] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:44:16,583] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:44:16,609] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:44:16,616] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.948053ms
[2022-09-12 05:44:16,642] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:44:16,668] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:44:16,725] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.343212ms
[2022-09-12 05:44:16,799] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:44:16,800] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.034431851795407, 'mem_free': 12965826560, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:44:16,800] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:44:16,804] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:16,805] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:17,422] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:17,427] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:17,427] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:18,402] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:18,409] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:18,410] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:19,474] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:19,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:19,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:20,569] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:26,997] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-12 05:44:27,163] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:44:28,336] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:44:28,364] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:44:28,364] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:44:28,425] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:44:28,478] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:44:28,531] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:44:28,532] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:44:28,611] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:44:28,612] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:44:28,637] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:28,765] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:44:28,765] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:28,792] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:44:28,817] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:44:28,817] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:28,843] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:44:28,868] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:44:28,869] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:28,895] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:44:28,920] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:44:28,920] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:28,947] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:44:28,947] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 05:44:28,947] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:44:28,948] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.748s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_9'}
[2022-09-12 05:44:29,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:29,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:29,639] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:29,672] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:44:29,755] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:44:29,755] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-12 05:44:29,756] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:44:30,062] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:44:30,089] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13072080896, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:44:30,117] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:44:30,118] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:44:30,118] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:44:30,157] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:44:30,190] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:44:30,191] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:44:30,220] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:44:30,220] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:44:30,221] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:44:30,221] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:44:30,270] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:44:30,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:30,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:30,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:30,911] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:44:31,026] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:44:31,027] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:44:31,057] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:44:31,057] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:44:31,176] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:44:31,203] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 13077245952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:44:31,229] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:44:31,256] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:44:31,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:44:31,308] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:44:31,311] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:31,311] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:31,909] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:31,910] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:44:32,029] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:44:32,030] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:44:32,059] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:44:32,060] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:44:32,173] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:44:32,201] - [task:165] INFO -  {'uptime': '753', 'memoryTotal': 15466930176, 'memoryFree': 13077446656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:44:32,228] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:44:32,256] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:44:32,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:44:32,309] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:44:32,314] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:32,314] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:32,914] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:32,915] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:44:33,033] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:44:33,034] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:44:33,063] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:44:33,064] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:44:33,178] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:44:33,204] - [task:165] INFO -  {'uptime': '754', 'memoryTotal': 15466930176, 'memoryFree': 13077270528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:44:33,230] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:44:33,258] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:44:33,258] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:44:33,310] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:44:33,313] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:33,313] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:33,924] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:33,925] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:44:34,056] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:44:34,058] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:44:34,087] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:44:34,088] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:44:34,180] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:44:34,849] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:44:39,854] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:44:39,950] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:44:39,954] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:39,954] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:40,576] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:40,577] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:44:40,696] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:44:40,697] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:44:40,698] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:44:41,328] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:44:41,385] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:44:41,386] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:44:42,313] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:44:42,633] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:44:42,918] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:44:42,921] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 05:44:42,980] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:42,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:43,606] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:43,610] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:43,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:44,499] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:44,511] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:44,511] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:45,623] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:45,631] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:45,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:46,723] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:44:53,294] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:44:53,294] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 26.68893074001364, 'mem_free': 13080227840, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:44:53,294] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:44:53,295] - [basetestcase:467] INFO - Time to execute basesetup : 24.262298107147217
[2022-09-12 05:44:53,351] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:44:53,351] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:44:53,408] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:44:53,408] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:44:53,464] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:44:53,465] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:44:53,521] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:44:53,521] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:44:53,577] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:44:53,644] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:44:53,644] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:44:53,644] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:44:58,654] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:44:58,657] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:44:58,658] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:44:59,280] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:00,419] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:45:00,592] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:45:02,729] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:45:02,808] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:02,808] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:45:02,809] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:45:32,839] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:45:32,869] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:45:32,894] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:45:32,961] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.697461ms
[2022-09-12 05:45:32,961] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4a64e6c0-8b35-473a-8168-a1e5d10b3852', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.697461ms', 'executionTime': '64.619385ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:45:32,961] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:45:32,986] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:45:33,014] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:45:33,662] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 645.624953ms
[2022-09-12 05:45:33,662] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:45:33,755] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:45:33,783] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:45:33,792] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.333193ms
[2022-09-12 05:45:34,019] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:34,052] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:45:34,070] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:45:34,070] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:45:34,081] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:45:34,139] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:34,956] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee015d97248ac24fc480cd5755bd052b83join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:45:34,981] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee015d97248ac24fc480cd5755bd052b83join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:45:35,018] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.6559ms
[2022-09-12 05:45:35,019] - [base_gsi:282] INFO - BUILD INDEX on default(`employee015d97248ac24fc480cd5755bd052b83join_yr`) USING GSI
[2022-09-12 05:45:36,049] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee015d97248ac24fc480cd5755bd052b83join_yr`) USING GSI
[2022-09-12 05:45:36,074] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee015d97248ac24fc480cd5755bd052b83join_yr%60%29+USING+GSI
[2022-09-12 05:45:36,097] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.17558ms
[2022-09-12 05:45:36,155] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015d97248ac24fc480cd5755bd052b83join_yr'
[2022-09-12 05:45:36,183] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015d97248ac24fc480cd5755bd052b83join_yr%27
[2022-09-12 05:45:36,193] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.462955ms
[2022-09-12 05:45:37,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015d97248ac24fc480cd5755bd052b83join_yr'
[2022-09-12 05:45:37,248] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015d97248ac24fc480cd5755bd052b83join_yr%27
[2022-09-12 05:45:37,258] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.498296ms
[2022-09-12 05:45:38,293] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015d97248ac24fc480cd5755bd052b83join_yr'
[2022-09-12 05:45:38,321] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015d97248ac24fc480cd5755bd052b83join_yr%27
[2022-09-12 05:45:38,331] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.660821ms
[2022-09-12 05:45:39,362] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:45:39,388] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 05:45:39,392] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.653997ms
[2022-09-12 05:45:39,393] - [base_gsi:504] INFO - {'requestID': '322ab3ba-a1f8-458f-813a-5024235e2498', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee015d97248ac24fc480cd5755bd052b83join_yr', 'index_id': 'e690dc7d4a87595f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.653997ms', 'executionTime': '2.57616ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 05:45:39,393] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:45:39,394] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:45:39,394] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:45:39,394] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:45:39,395] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:45:39,396] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:45:39,396] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:45:39,396] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:45:39,456] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:45:39,496] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:45:39,523] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 05:45:39,681] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 144.015411ms
[2022-09-12 05:45:39,681] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:45:39,682] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:45:41,919] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:41,945] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee015d97248ac24fc480cd5755bd052b83join_yr'
[2022-09-12 05:45:41,975] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee015d97248ac24fc480cd5755bd052b83join_yr%27
[2022-09-12 05:45:41,983] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.742859ms
[2022-09-12 05:45:41,983] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '73ae81a9-7312-4693-b70e-7be62d9d68bc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.742859ms', 'executionTime': '5.673742ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:45:42,084] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:42,087] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:42,087] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:42,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:42,831] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:45:42,831] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:45:42,938] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:45:42,997] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:43,002] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:43,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:43,637] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:43,692] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:45:43,692] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:45:43,799] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:45:43,854] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:45:43,854] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:45:43,854] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:45:43,880] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:45:43,905] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:45:43,913] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.931882ms
[2022-09-12 05:45:43,913] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f3f1f5e5-5597-4cd8-83fc-0a20dfd5ec58', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.931882ms', 'executionTime': '5.869986ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:45:43,967] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:45:43,967] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 16.41860197197031, 'mem_free': 12912959488, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:45:43,967] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:45:43,971] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:43,971] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:44,598] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:44,603] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:44,603] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:45,249] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:45,254] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:45,254] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:46,286] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:46,295] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:46,295] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:47,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:54,746] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-12 05:45:54,881] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:45:54,935] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:45:54,988] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:45:55,041] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:45:55,041] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:45:55,124] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:45:55,125] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:45:55,150] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:55,279] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:45:55,279] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:55,307] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:45:55,332] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:45:55,333] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:55,359] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:45:55,385] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:45:55,385] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:55,412] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:45:55,437] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:45:55,438] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:55,463] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:45:55,464] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 05:45:55,464] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:45:55,465] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 86.485s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_10'}
[2022-09-12 05:45:55,544] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:55,544] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:56,218] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:56,252] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:45:56,335] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:45:56,335] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-12 05:45:56,336] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:45:56,572] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:45:56,599] - [task:165] INFO -  {'uptime': '840', 'memoryTotal': 15466930176, 'memoryFree': 13067718656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:45:56,625] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:45:56,625] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:45:56,626] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:45:56,662] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:45:56,696] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:45:56,697] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:45:56,725] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:45:56,726] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:45:56,726] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:45:56,726] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:45:56,775] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:45:56,778] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:56,778] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:57,463] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:57,464] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:45:57,591] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:45:57,592] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:45:57,622] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:45:57,623] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:45:57,744] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:45:57,771] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 13069238272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:45:57,797] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:45:57,825] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:45:57,826] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:45:57,877] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:45:57,883] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:57,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:58,540] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:58,541] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:45:58,666] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:45:58,667] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:45:58,700] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:45:58,701] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:45:58,817] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:45:58,843] - [task:165] INFO -  {'uptime': '839', 'memoryTotal': 15466930176, 'memoryFree': 13068886016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:45:58,869] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:45:58,898] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:45:58,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:45:58,949] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:45:58,953] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:45:58,953] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:45:59,619] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:45:59,620] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:45:59,746] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:45:59,747] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:45:59,777] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:45:59,778] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:45:59,898] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:45:59,926] - [task:165] INFO -  {'uptime': '839', 'memoryTotal': 15466930176, 'memoryFree': 13069082624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:45:59,951] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:45:59,980] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:45:59,980] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:46:00,033] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:46:00,036] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:00,037] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:00,701] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:00,702] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:46:00,829] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:46:00,830] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:46:00,859] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:46:00,859] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:46:00,950] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:46:01,634] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:46:06,638] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:46:06,726] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:46:06,730] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:06,730] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:07,445] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:07,446] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:46:07,579] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:46:07,580] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:46:07,581] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:46:08,136] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:46:08,191] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:46:08,191] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:46:09,093] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:46:09,408] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:46:09,672] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:46:09,675] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 05:46:09,731] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:09,731] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:10,407] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:10,413] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:10,413] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:11,430] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:11,441] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:11,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:12,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:12,582] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:12,582] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:13,775] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:20,417] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:46:20,418] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.6791304143477, 'mem_free': 13067685888, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:46:20,418] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:46:20,418] - [basetestcase:467] INFO - Time to execute basesetup : 24.87657642364502
[2022-09-12 05:46:20,469] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:46:20,469] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:46:20,521] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:46:20,521] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:46:20,573] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:46:20,574] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:46:20,626] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:46:20,627] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:46:20,678] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:46:20,740] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:46:20,740] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:46:20,741] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:46:25,754] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:46:25,757] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:46:25,757] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:46:26,406] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:46:27,484] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:46:27,767] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:46:30,833] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:46:30,912] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:46:30,912] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:46:30,912] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:47:00,943] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:47:00,973] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:47:00,998] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:47:01,067] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.511068ms
[2022-09-12 05:47:01,067] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c236d715-6a38-4961-841b-32b17ea608ea', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.511068ms', 'executionTime': '66.438321ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:47:01,067] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:47:01,093] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:47:01,118] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:47:01,761] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 640.701415ms
[2022-09-12 05:47:01,761] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:47:01,816] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:47:01,856] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:47:01,865] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.065289ms
[2022-09-12 05:47:02,071] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:02,110] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:47:02,128] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:47:02,129] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:47:02,147] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:47:02,225] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:03,032] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:47:03,057] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:47:03,103] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.136247ms
[2022-09-12 05:47:03,103] - [base_gsi:282] INFO - BUILD INDEX on default(`employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr`) USING GSI
[2022-09-12 05:47:04,134] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr`) USING GSI
[2022-09-12 05:47:04,159] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%60%29+USING+GSI
[2022-09-12 05:47:04,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.87118ms
[2022-09-12 05:47:04,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr'
[2022-09-12 05:47:04,258] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%27
[2022-09-12 05:47:04,266] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.6378ms
[2022-09-12 05:47:05,297] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr'
[2022-09-12 05:47:05,322] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%27
[2022-09-12 05:47:05,332] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.564929ms
[2022-09-12 05:47:06,364] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr'
[2022-09-12 05:47:06,390] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%27
[2022-09-12 05:47:06,397] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.56955ms
[2022-09-12 05:47:07,428] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:47:07,454] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 05:47:07,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.625555ms
[2022-09-12 05:47:07,458] - [base_gsi:504] INFO - {'requestID': '573f4c89-fef2-4304-a830-6c6bbee962a2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr', 'index_id': '6a9d1c8b4900b37c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.625555ms', 'executionTime': '2.568546ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 05:47:07,459] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:47:07,459] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:07,459] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:47:07,459] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:47:07,460] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:07,460] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:07,460] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:07,460] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:07,519] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:07,557] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:47:07,582] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 05:47:07,769] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 174.207018ms
[2022-09-12 05:47:07,769] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:47:07,770] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:47:08,962] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-12 05:47:09,014] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-12 05:47:09,047] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:47:09,076] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 05:47:09,080] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.978836ms
[2022-09-12 05:47:09,080] - [base_gsi:504] INFO - {'requestID': '9d7ba540-02e9-4ad2-9a20-fe14fdea6616', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr', 'index_id': '6a9d1c8b4900b37c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.978836ms', 'executionTime': '1.911326ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 05:47:09,080] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:47:09,081] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:09,081] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:47:09,081] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:47:09,081] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:09,081] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:09,081] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:09,082] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:47:09,134] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:47:09,175] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:47:09,201] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 05:47:09,307] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.707658ms
[2022-09-12 05:47:09,307] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:47:09,307] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:47:10,506] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr ON default USING GSI
[2022-09-12 05:47:10,532] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr+ON+default+USING+GSI
[2022-09-12 05:47:10,573] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.160501ms
[2022-09-12 05:47:10,606] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr'
[2022-09-12 05:47:10,632] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeead8c9ad63e264827a6c68bb93f44a52fjoin_yr%27
[2022-09-12 05:47:10,639] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.653203ms
[2022-09-12 05:47:10,639] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0ebbb37f-e7b3-4a02-aa74-7cd0c5dbb395', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.653203ms', 'executionTime': '5.59564ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:47:10,740] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:10,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:10,743] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:11,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:11,497] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:47:11,497] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:47:11,609] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:47:11,667] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:11,670] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:11,670] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:12,338] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:12,393] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:47:12,393] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:47:12,508] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:47:12,564] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:12,564] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:47:12,564] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:47:12,589] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:47:12,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:47:12,624] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.591745ms
[2022-09-12 05:47:12,653] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:47:12,679] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:47:12,728] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.093746ms
[2022-09-12 05:47:12,808] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:47:12,808] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.8462235256481, 'mem_free': 12917051392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:47:12,808] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:47:12,811] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:12,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:13,502] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:13,507] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:13,507] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:14,215] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:14,220] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:14,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:15,256] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:15,264] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:15,264] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:16,472] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:24,292] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-12 05:47:24,433] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:47:25,677] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:47:25,706] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:47:25,706] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:47:25,767] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:47:25,820] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:47:25,872] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:47:25,873] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:47:25,952] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:47:25,953] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:47:25,978] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:26,106] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:47:26,106] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:26,133] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:47:26,159] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:47:26,159] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:26,187] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:47:26,213] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:47:26,213] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:26,240] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:47:26,265] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:47:26,265] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:26,292] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:47:26,293] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 05:47:26,293] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:47:26,294] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 90.804s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_11'}
[2022-09-12 05:47:26,372] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:26,373] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:27,119] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:27,156] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:47:27,238] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:47:27,239] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-12 05:47:27,239] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:47:27,400] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:47:27,429] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 13061398528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:47:27,457] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:47:27,458] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:47:27,458] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:47:27,497] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:47:27,535] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:47:27,535] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:47:27,565] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:47:27,566] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:47:27,566] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:47:27,567] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:47:27,614] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:47:27,617] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:27,618] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:28,331] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:28,332] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:47:28,463] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:47:28,464] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:47:28,494] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:47:28,495] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:47:28,619] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:47:28,646] - [task:165] INFO -  {'uptime': '933', 'memoryTotal': 15466930176, 'memoryFree': 13058215936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:47:28,671] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:47:28,698] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:47:28,699] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:47:28,749] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:47:28,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:28,753] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:29,445] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:29,446] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:47:29,584] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:47:29,585] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:47:29,614] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:47:29,615] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:47:29,738] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:47:29,771] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 13058080768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:47:29,798] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:47:29,826] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:47:29,826] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:47:29,876] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:47:29,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:29,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:30,600] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:30,601] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:47:30,749] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:47:30,750] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:47:30,781] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:47:30,782] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:47:30,903] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:47:30,930] - [task:165] INFO -  {'uptime': '929', 'memoryTotal': 15466930176, 'memoryFree': 13060411392, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:47:30,956] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:47:30,984] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:47:30,984] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:47:31,035] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:47:31,037] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:31,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:31,723] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:31,724] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:47:31,851] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:47:31,852] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:47:31,882] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:47:31,883] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:47:31,970] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:47:32,513] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:47:37,516] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:47:37,599] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:47:37,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:37,605] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:38,264] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:38,265] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:47:38,394] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:47:38,395] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:47:38,396] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:47:39,082] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:47:39,138] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:47:39,139] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:47:40,045] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:47:40,345] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:47:40,602] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:47:40,605] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 05:47:40,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:40,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:41,323] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:41,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:41,328] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:42,344] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:42,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:42,356] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:43,537] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:43,549] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:43,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:44,753] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:51,899] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:47:51,899] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.68924658560258, 'mem_free': 13060210688, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:47:51,899] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:47:51,900] - [basetestcase:467] INFO - Time to execute basesetup : 25.529454946517944
[2022-09-12 05:47:51,949] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:47:51,949] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:47:52,002] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:47:52,002] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:47:52,055] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:47:52,055] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:47:52,108] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:47:52,108] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:47:52,159] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:47:52,219] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:47:52,220] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:47:52,220] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:47:57,237] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:47:57,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:47:57,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:47:57,918] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:47:59,091] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:47:59,263] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:48:02,358] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:48:02,439] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:02,439] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:48:02,440] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:48:32,463] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:48:32,493] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:48:32,518] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:48:32,589] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.815736ms
[2022-09-12 05:48:32,589] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8a396425-1695-4f47-915d-30ffd6213e42', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.815736ms', 'executionTime': '68.757126ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:48:32,590] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:48:32,615] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:48:32,640] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:48:33,354] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 711.512015ms
[2022-09-12 05:48:33,354] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:48:33,427] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:48:33,471] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:48:33,480] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.597421ms
[2022-09-12 05:48:33,692] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:33,734] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:48:33,758] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:48:33,759] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:48:33,782] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:48:33,845] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:34,656] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee69648dc859ea41aca1de93ef329e8802join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 05:48:34,686] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee69648dc859ea41aca1de93ef329e8802join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 05:48:34,734] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.10188ms
[2022-09-12 05:48:34,736] - [base_gsi:282] INFO - BUILD INDEX on default(`employee69648dc859ea41aca1de93ef329e8802join_yr`) USING GSI
[2022-09-12 05:48:35,766] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee69648dc859ea41aca1de93ef329e8802join_yr`) USING GSI
[2022-09-12 05:48:35,792] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee69648dc859ea41aca1de93ef329e8802join_yr%60%29+USING+GSI
[2022-09-12 05:48:35,820] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.955825ms
[2022-09-12 05:48:35,862] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee69648dc859ea41aca1de93ef329e8802join_yr'
[2022-09-12 05:48:35,892] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee69648dc859ea41aca1de93ef329e8802join_yr%27
[2022-09-12 05:48:35,901] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.841977ms
[2022-09-12 05:48:36,932] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee69648dc859ea41aca1de93ef329e8802join_yr'
[2022-09-12 05:48:36,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee69648dc859ea41aca1de93ef329e8802join_yr%27
[2022-09-12 05:48:36,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.786383ms
[2022-09-12 05:48:37,997] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee69648dc859ea41aca1de93ef329e8802join_yr'
[2022-09-12 05:48:38,022] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee69648dc859ea41aca1de93ef329e8802join_yr%27
[2022-09-12 05:48:38,032] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.049392ms
[2022-09-12 05:48:39,062] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:48:39,088] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 05:48:39,092] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.771927ms
[2022-09-12 05:48:39,092] - [base_gsi:504] INFO - {'requestID': '74544665-7514-46d5-9892-0777e52af618', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee69648dc859ea41aca1de93ef329e8802join_yr', 'index_id': 'ae4697271fc61ed6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.771927ms', 'executionTime': '2.690839ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 05:48:39,092] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 05:48:39,093] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:48:39,093] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 05:48:39,093] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 05:48:39,094] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:48:39,094] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:48:39,094] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:48:39,094] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 05:48:39,155] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 05:48:39,199] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 05:48:39,224] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 05:48:39,438] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 200.980314ms
[2022-09-12 05:48:39,438] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 05:48:39,439] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 05:48:41,658] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 05:48:44,111] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:48:44,170] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:48:44,171] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:48:45,065] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:48:45,375] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:48:45,654] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:48:45,658] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 05:48:47,695] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:48:47,695] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:48:47,723] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:48:47,753] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:48:47,754] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:48:47,783] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:48:47,813] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:48:47,813] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:48:47,841] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:48:47,869] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:48:47,869] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:48:47,896] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:48:47,946] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:47,970] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee69648dc859ea41aca1de93ef329e8802join_yr'
[2022-09-12 05:48:47,997] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee69648dc859ea41aca1de93ef329e8802join_yr%27
[2022-09-12 05:48:48,043] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.780172ms
[2022-09-12 05:48:48,044] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c28363ec-4c06-48e7-8839-9dfedde4806e', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '44.780172ms', 'executionTime': '44.711177ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:48:48,070] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee69648dc859ea41aca1de93ef329e8802join_yr'
[2022-09-12 05:48:48,096] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee69648dc859ea41aca1de93ef329e8802join_yr%27
[2022-09-12 05:48:48,099] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.390041ms
[2022-09-12 05:48:48,099] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '345411dc-78c5-436e-a206-13f639287a7b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.390041ms', 'executionTime': '1.313487ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:48:48,154] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:48,288] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:48,291] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:48,291] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:48,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:48:49,050] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:48:49,050] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:48:49,164] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:48:49,221] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:49,225] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:49,226] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:49,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:48:49,966] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:48:49,966] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:48:50,087] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:48:50,152] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:48:50,153] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:48:50,153] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:48:50,179] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:48:50,204] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:48:50,214] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.352999ms
[2022-09-12 05:48:50,214] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '01bec2ac-5236-423f-b6d0-4a38d856173c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '8.352999ms', 'executionTime': '8.272383ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:48:50,214] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:48:50,240] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:48:50,265] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:48:50,267] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.018161ms
[2022-09-12 05:48:50,267] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c727d98b-bebf-48d1-89c3-9d08bec8d33b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.018161ms', 'executionTime': '974.653┬Ás', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:48:50,317] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:48:50,317] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.18099233529004, 'mem_free': 12880605184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:48:50,317] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:48:50,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:50,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:51,034] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:48:51,039] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:51,039] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:51,839] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:48:51,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:51,847] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:53,062] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:48:53,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:48:53,070] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:48:54,272] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:01,746] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 05:49:01,891] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:49:02,200] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:49:02,229] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:49:02,229] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:49:02,287] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:49:02,342] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:49:02,395] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:49:02,396] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:49:02,475] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:49:02,476] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:49:02,502] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:02,642] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:49:02,642] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:02,672] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:49:02,702] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:49:02,702] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:02,731] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:49:02,763] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:49:02,764] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:02,793] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:49:02,821] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:49:02,821] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:02,849] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:49:02,849] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 05:49:02,849] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:49:02,850] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 96.536s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_05-31-48/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-12 05:49:02,913] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:02,913] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:03,700] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:03,733] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:49:03,819] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 05:49:03,820] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-12 05:49:03,820] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 05:49:03,941] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 05:49:03,973] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 13047930880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:49:04,002] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 05:49:04,002] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 05:49:04,003] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 05:49:04,044] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 05:49:04,078] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 05:49:04,079] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 05:49:04,107] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 05:49:04,108] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 05:49:04,108] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 05:49:04,109] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 05:49:04,157] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:49:04,160] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:04,160] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:04,898] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:04,899] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:49:05,031] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:49:05,032] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:49:05,064] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:49:05,065] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:49:05,189] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 05:49:05,219] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 13049053184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:49:05,244] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:49:05,272] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 05:49:05,273] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 05:49:05,324] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:49:05,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:05,327] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:06,004] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:06,005] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:49:06,141] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:49:06,142] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:49:06,173] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:49:06,174] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:49:06,293] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 05:49:06,322] - [task:165] INFO -  {'uptime': '1029', 'memoryTotal': 15466930176, 'memoryFree': 13039267840, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:49:06,350] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:49:06,379] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 05:49:06,380] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 05:49:06,434] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:49:06,437] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:06,437] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:07,170] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:07,171] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:49:07,309] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:49:07,311] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:49:07,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:49:07,342] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:49:07,461] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 05:49:07,488] - [task:165] INFO -  {'uptime': '1030', 'memoryTotal': 15466930176, 'memoryFree': 13047287808, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 05:49:07,514] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 05:49:07,542] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 05:49:07,542] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 05:49:07,593] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 05:49:07,596] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:07,596] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:08,342] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:08,343] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:49:08,484] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:49:08,486] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:49:08,516] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 05:49:08,517] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-12 05:49:08,611] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 05:49:09,216] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:49:14,222] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 05:49:14,304] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 05:49:14,307] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:14,307] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:15,044] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:15,045] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 05:49:15,181] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 05:49:15,182] - [remote_util:5249] INFO - b'ok'
[2022-09-12 05:49:15,183] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 05:49:15,722] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 05:49:15,781] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 05:49:15,782] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 05:49:16,692] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:49:17,003] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:49:17,269] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 05:49:17,272] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-12 05:49:17,325] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:17,325] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:18,033] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:18,037] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:18,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:19,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:19,105] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:19,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:20,318] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:20,329] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:20,330] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:21,549] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:28,861] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:49:28,861] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.91897475838509, 'mem_free': 13044088832, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:49:28,861] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:49:28,861] - [basetestcase:467] INFO - Time to execute basesetup : 25.951672792434692
[2022-09-12 05:49:28,911] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:49:28,912] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:49:28,962] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:49:28,963] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:49:29,015] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:49:29,016] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:49:29,068] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 05:49:29,068] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 05:49:29,117] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:49:29,183] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-12 05:49:29,183] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 05:49:29,185] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 05:49:34,197] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 05:49:34,200] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:49:34,200] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:49:34,896] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:49:36,075] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 05:49:36,256] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 05:49:38,435] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 05:49:38,511] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:49:38,512] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 05:49:38,512] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 05:50:08,535] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 05:50:08,564] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:50:08,589] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:50:08,657] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.984537ms
[2022-09-12 05:50:08,657] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e5c67022-e1fd-45f6-9696-841deb67175c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.984537ms', 'executionTime': '65.921442ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 05:50:08,657] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 05:50:08,683] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 05:50:08,708] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 05:50:09,359] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 649.808172ms
[2022-09-12 05:50:09,360] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 05:50:09,428] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:50:09,462] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:50:09,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.844824ms
[2022-09-12 05:50:09,672] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:50:09,708] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 05:50:09,723] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 05:50:09,724] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 05:50:09,738] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 05:50:09,811] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:50:09,952] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:50:09,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:09,956] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:10,767] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:10,822] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:50:10,823] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 05:50:10,949] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:50:11,014] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:50:11,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:11,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:11,778] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:11,833] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 05:50:11,833] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 05:50:11,961] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 05:50:12,019] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 05:50:12,020] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 05:50:12,020] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:50:12,046] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 05:50:12,071] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 05:50:12,081] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.340413ms
[2022-09-12 05:50:12,107] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 05:50:12,132] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 05:50:12,185] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.714616ms
[2022-09-12 05:50:12,248] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 05:50:12,249] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.036181044407591, 'mem_free': 12874698752, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 05:50:12,249] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 05:50:12,253] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:12,253] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:13,060] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:13,065] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:13,065] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:14,333] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:14,341] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:14,341] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:15,722] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:15,736] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 05:50:15,736] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 05:50:17,150] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 05:50:24,693] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-12 05:50:24,842] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 05:50:25,273] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 05:50:25,301] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 05:50:25,301] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 05:50:25,357] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:50:25,408] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:50:25,460] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 05:50:25,461] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 05:50:25,543] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 05:50:25,544] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 05:50:25,569] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:50:25,697] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 05:50:25,698] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:50:25,724] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 05:50:25,749] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 05:50:25,749] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:50:25,775] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 05:50:25,801] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 05:50:25,802] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:50:25,828] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 05:50:25,853] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 05:50:25,853] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 05:50:25,879] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 05:50:25,880] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-12 05:50:25,880] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 05:50:25,881] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 83.051s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91:  7430 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-12.09.2022-04.04.cfg
Build Log: make-12.09.2022-04.04.log
Server Log: logs-12.09.2022-04.04.tar.gz

Finished