Building

Started building at 2022/09/12 06:01:49
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 12.09.2022-06.01
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at fb7c5c5, changes since last good build: 
fatal: Invalid revision range 1c5995aedb73edda33523300ee73b3da1e3ca5a2..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable fdef834d] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 1deae03f] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable c34bcabd] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/12 06:51:40
Testing mode: sanity,unit,functional,integration
Using storage type: fdb
Setting ulimit to 200000

Simple Test

Sep 12 06:56:44 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 07:00:40 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 07:01:24 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 12 07:02:51 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 12 07:03:44 b" 'stop_on_failure': 'True'}"
Sep 12 07:03:44 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 07:03:44 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_1'}"
Sep 12 07:03:44 b'-->result: '
Sep 12 07:03:44 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 12 07:03:44 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 07:03:44 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_2'}"
Sep 12 07:03:44 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:03:44 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 12 07:03:44 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_3'}"
Sep 12 07:03:44 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:03:44 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:03:44 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 12 07:03:44 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_4'}"
Sep 12 07:03:44 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:03:44 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:03:44 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 07:13:31 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 12 07:14:18 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 12 07:18:03 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 07:22:00 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 07:24:42 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 07:24:42 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_5'}"
Sep 12 07:24:42 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 12 07:24:42 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 12 07:24:42 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_6'}"
Sep 12 07:24:42 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 07:24:42 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 12 07:24:42 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_7'}"
Sep 12 07:24:42 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 12 07:24:42 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 12 07:24:42 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_06-52-01/test_8'}"
Sep 12 07:24:42 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 07:24:42 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 07:24:42 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 12 07:24:42 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 12 07:24:43 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 07:24:43 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 07:24:43 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 12 07:24:43 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 12 07:24:43 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 12 07:24:43 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 12 07:24:43 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 12 07:24:43 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=forestdb
Printing gsi_type=forestdb
gsi_type=forestdb
In here
-p makefile=True,gsi_type=forestdb
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'forestdb',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-12 07:27:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:27:34 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:27:34 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:27:34 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 07:27:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 07:27:43 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:27:56 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:27:56 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:27:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:27:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:27:57 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:27:57 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:02 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 07:28:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 07:28:02 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:28:02 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9003/pools/default error [Errno 111] Connection refused 
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 07:28:05 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-12 07:28:06 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '27', 'memoryTotal': 15466930176, 'memoryFree': 2608123904, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:28:06 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 07:28:06 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:28:07 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:28:07 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 13570289664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:28:07 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 07:28:07 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:28:08 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:28:08 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 2305810432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:28:08 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:08 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:28:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:28:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 2202529792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:28:09 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:09 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:28:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:28:10 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:28:10 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 07:28:10 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 07:28:10 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-12 07:28:10 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 07:28:15 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 07:28:15 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-12 07:28:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:15 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:16 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:28:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:28:16 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:28:16 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 07:28:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 07:28:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 07:28:16 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 07:28:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:28:17 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:28:18 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:18 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 36.45773407391828, 'mem_free': 13473849344, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 46.65720868110657
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 07:28:21 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 07:28:26 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 07:28:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:28:26 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:28:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:28:27 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 07:28:28 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:28:29 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 07:28:29 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:28:29 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 07:28:29 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 07:28:59 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 07:28:59 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:28:59 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 151.581618ms
2022-09-12 07:29:00 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '0bc078a1-01a7-4101-ba83-3c54e9ffa277', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '151.581618ms', 'executionTime': '151.363445ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 756.913769ms
2022-09-12 07:29:00 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.090555ms
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.6786ms
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 47.796442ms
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.977467826702027, 'mem_free': 13407387648, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:02 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:03 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:06 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 07:29:06 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 07:29:07 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 07:29:07 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 93.031s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-12 07:29:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:07 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:08 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.46456760598927, 'mem_free': 13383221248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:09 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 07:29:13 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 07:29:13 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 07:29:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13383221248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 07:29:15 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:29:15 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 07:29:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '94', 'memoryTotal': 15466930176, 'memoryFree': 13376716800, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:29:16 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13377634304, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:29:16 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:29:16 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '90', 'memoryTotal': 15466930176, 'memoryFree': 13383434240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 07:29:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=forestdb
2022-09-12 07:29:17 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
2022-09-12 07:29:17 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 07:29:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 07:29:23 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 07:29:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 07:29:23 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-12 07:29:23 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 07:29:24 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:29:25 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:29:25 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:25 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:26 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:26 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 30.76836952960515, 'mem_free': 13364817920, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 21.65600275993347
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'forestdb'} set
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 07:29:29 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 07:29:34 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 07:29:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:29:34 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:29:35 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:29:35 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 07:29:36 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 07:29:37 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 07:29:37 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:29:37 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 07:29:37 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 07:30:07 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 71.105426ms
2022-09-12 07:30:08 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': '16e69cc1-4228-4a28-b660-4cf46a78a20a', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.105426ms', 'executionTime': '70.974192ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 712.67834ms
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:30:08 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.845446ms
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 07:30:09 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:30:10 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-12 07:30:10 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-12 07:30:10 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.925691ms
2022-09-12 07:30:10 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title) USING GSI
2022-09-12 07:30:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title) USING GSI
2022-09-12 07:30:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%29+USING+GSI
2022-09-12 07:30:11 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 21.814973ms
2022-09-12 07:30:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title'
2022-09-12 07:30:12 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%27
2022-09-12 07:30:12 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.853077ms
2022-09-12 07:30:13 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title'
2022-09-12 07:30:13 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%27
2022-09-12 07:30:13 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.392956ms
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 3.422463ms
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '369a508b-43a4-4889-a775-d7d49361f5ae', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title', 'index_id': '7277e6600733aa2f', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.422463ms', 'executionTime': '3.300861ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:14 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-12 07:30:14 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 154.17503ms
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-12 07:30:15 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-12 07:30:16 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:16 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title'
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%27
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.739023ms
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title ON default USING GSI
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title+ON+default+USING+GSI
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 39.345885ms
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title'
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb5e2761ddf5c4f40a7cdc2eed6fe5f58job_title%27
2022-09-12 07:30:17 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.820709ms
2022-09-12 07:30:17 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'fdb9f6e7-fbba-4537-9206-0387b3fe5176', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.820709ms', 'executionTime': '5.69212ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 07:30:17 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:30:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:17 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.062138ms
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 68.178615ms
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.212075764983741, 'mem_free': 13293867008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:18 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:19 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 07:30:20 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 07:30:20 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 07:30:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 07:30:24 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 07:30:25 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 07:30:25 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 78.059s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_2'}
[2022-09-12 07:30:25,939] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:25,939] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:26,195] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:26,227] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:30:26,307] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:30:26,307] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-12 07:30:26,308] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:30:26,969] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:30:26,999] - [task:165] INFO -  {'uptime': '168', 'memoryTotal': 15466930176, 'memoryFree': 13253722112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:30:27,027] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:30:27,027] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:30:27,028] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:30:27,069] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:30:27,105] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:30:27,105] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:30:27,139] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:30:27,140] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:30:27,140] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:30:27,141] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:30:27,193] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:30:27,196] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:27,196] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:27,486] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:27,487] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:30:27,560] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:30:27,561] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:30:27,591] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:30:27,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:30:27,618] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:30:27,705] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:30:27,734] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13285502976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:30:27,760] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:30:27,790] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:30:27,790] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:30:27,841] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:30:27,845] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:27,846] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:28,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:28,098] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:30:28,165] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:30:28,166] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:30:28,195] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:30:28,196] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:30:28,222] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:30:28,306] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:30:28,332] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13285609472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:30:28,357] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:30:28,386] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:30:28,386] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:30:28,436] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:30:28,439] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:28,439] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:28,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:28,690] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:30:28,758] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:30:28,759] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:30:28,788] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:30:28,788] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:30:28,816] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:30:28,901] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:30:28,928] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13285785600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:30:28,953] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:30:28,981] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:30:28,982] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:30:29,032] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:30:29,035] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:29,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:29,304] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:29,305] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:30:29,371] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:30:29,371] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:30:29,401] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:30:29,402] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:30:29,428] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:30:29,483] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:30:30,139] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:30:35,144] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:30:35,230] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:30:35,235] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:35,235] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:35,513] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:35,514] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:30:35,581] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:30:35,582] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:30:35,583] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:30:36,617] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:30:36,671] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:30:36,671] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:30:37,580] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:30:37,898] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:30:38,171] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:30:38,227] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:30:38,283] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:38,283] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:38,540] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:38,545] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:38,545] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:38,793] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:38,798] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:38,799] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:39,056] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:39,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:39,061] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:39,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:43,088] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:30:43,088] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.34342729835319, 'mem_free': 13284511744, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:30:43,088] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:30:43,089] - [basetestcase:467] INFO - Time to execute basesetup : 17.151343822479248
[2022-09-12 07:30:43,139] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:30:43,139] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:30:43,192] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:30:43,192] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:30:43,246] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:30:43,246] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:30:43,298] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:30:43,299] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:30:43,348] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:30:43,408] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:30:43,409] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:30:43,409] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:30:48,421] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:30:48,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:30:48,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:30:48,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:30:49,627] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:30:49,796] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:30:51,445] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:30:51,467] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 07:30:52,124] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:30:52,124] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:30:52,125] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:31:22,152] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:31:22,181] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:31:22,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:31:22,281] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.286275ms
[2022-09-12 07:31:22,281] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '90a5f88b-90ec-4991-94a7-511a74a54480', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.286275ms', 'executionTime': '70.154429ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:31:22,281] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:31:22,309] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:31:22,337] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:31:23,073] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 733.959063ms
[2022-09-12 07:31:23,073] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:31:23,140] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:31:23,178] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:31:23,186] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.764428ms
[2022-09-12 07:31:23,394] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:31:23,432] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:31:23,447] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:31:23,447] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:31:23,463] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:31:23,537] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:31:24,354] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec412f5a9607e455e8055519880e12d2ejob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:31:24,380] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec412f5a9607e455e8055519880e12d2ejob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:31:24,436] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.197089ms
[2022-09-12 07:31:24,437] - [base_gsi:282] INFO - BUILD INDEX on default(employeec412f5a9607e455e8055519880e12d2ejob_title) USING GSI
[2022-09-12 07:31:25,467] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec412f5a9607e455e8055519880e12d2ejob_title) USING GSI
[2022-09-12 07:31:25,493] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec412f5a9607e455e8055519880e12d2ejob_title%29+USING+GSI
[2022-09-12 07:31:25,517] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.418441ms
[2022-09-12 07:31:26,548] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec412f5a9607e455e8055519880e12d2ejob_title'
[2022-09-12 07:31:26,573] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec412f5a9607e455e8055519880e12d2ejob_title%27
[2022-09-12 07:31:26,582] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.508306ms
[2022-09-12 07:31:27,614] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec412f5a9607e455e8055519880e12d2ejob_title'
[2022-09-12 07:31:27,641] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec412f5a9607e455e8055519880e12d2ejob_title%27
[2022-09-12 07:31:27,651] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.246093ms
[2022-09-12 07:31:28,098] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 07:31:28,267] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:31:29,152] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:31:29,653] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:29,688] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:31:29,723] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 07:31:29,727] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.66532ms
[2022-09-12 07:31:29,728] - [task:3246] INFO - {'requestID': '7825040c-3551-4784-8930-f19114de0d8a', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec412f5a9607e455e8055519880e12d2ejob_title', 'index_id': '5e739a2e6b48b5bf', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.66532ms', 'executionTime': '2.589634ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 07:31:29,728] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:29,728] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:29,729] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:31:29,729] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:31:29,730] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 07:31:29,730] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:31:29,730] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:31:29,731] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:31:29,731] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:31:30,730] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:30,762] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:31:30,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 07:31:30,922] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 123.28856ms
[2022-09-12 07:31:30,922] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:31:30,922] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:31:31,681] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:31,681] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:31:32,713] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec412f5a9607e455e8055519880e12d2ejob_title'
[2022-09-12 07:31:32,739] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec412f5a9607e455e8055519880e12d2ejob_title%27
[2022-09-12 07:31:32,751] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.962658ms
[2022-09-12 07:31:32,780] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec412f5a9607e455e8055519880e12d2ejob_title ON default USING GSI
[2022-09-12 07:31:32,808] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec412f5a9607e455e8055519880e12d2ejob_title+ON+default+USING+GSI
[2022-09-12 07:31:32,860] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.546684ms
[2022-09-12 07:31:32,896] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec412f5a9607e455e8055519880e12d2ejob_title'
[2022-09-12 07:31:32,922] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec412f5a9607e455e8055519880e12d2ejob_title%27
[2022-09-12 07:31:32,930] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.862011ms
[2022-09-12 07:31:32,930] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '99b7d859-9312-462b-a035-ca3f61f39fe3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.862011ms', 'executionTime': '5.799531ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:31:33,043] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:31:33,046] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:33,046] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:33,425] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:33,482] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:31:33,483] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:31:33,546] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:31:33,602] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:31:33,605] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:33,606] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:33,974] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:34,033] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:31:34,033] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:31:34,109] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:31:34,165] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:31:34,165] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:31:34,165] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:31:34,192] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:31:34,218] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:31:34,228] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.221403ms
[2022-09-12 07:31:34,257] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:31:34,284] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:31:34,327] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.800815ms
[2022-09-12 07:31:34,406] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:31:34,406] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.98651458091252, 'mem_free': 13168181248, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:31:34,406] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:31:34,411] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:34,411] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:34,808] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:34,814] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:34,814] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:35,202] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:35,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:35,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:35,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:35,876] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:35,877] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:36,495] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:40,433] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:31:40,680] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:31:41,311] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:31:41,339] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:31:41,339] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:31:41,396] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:31:41,449] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:31:41,505] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:31:41,506] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:31:41,588] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:31:41,589] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:31:41,617] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:41,758] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:31:41,759] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:41,787] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:31:41,832] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:31:41,832] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:41,860] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:31:41,886] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:31:41,887] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:41,913] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:31:41,940] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:31:41,940] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:41,967] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:31:41,967] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:31:41,967] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:31:41,968] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 76.084s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_3'}
[2022-09-12 07:31:42,053] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:42,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:42,368] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:42,399] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:31:42,478] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:31:42,478] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-12 07:31:42,479] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:31:43,082] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:31:43,110] - [task:165] INFO -  {'uptime': '244', 'memoryTotal': 15466930176, 'memoryFree': 13224308736, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:31:43,135] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:31:43,135] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:31:43,136] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:31:43,171] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:31:43,215] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:31:43,216] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:31:43,246] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:31:43,247] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:31:43,247] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:31:43,248] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:31:43,299] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:31:43,302] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:43,303] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:43,633] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:43,634] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:31:43,714] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:31:43,716] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:31:43,746] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:31:43,747] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:31:43,773] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:31:43,862] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:31:43,891] - [task:165] INFO -  {'uptime': '240', 'memoryTotal': 15466930176, 'memoryFree': 13159317504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:31:43,917] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:31:43,945] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:31:43,946] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:31:43,999] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:31:44,003] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:44,004] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:44,320] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:44,321] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:31:44,397] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:31:44,398] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:31:44,429] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:31:44,430] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:31:44,455] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:31:44,541] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:31:44,568] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13169098752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:31:44,600] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:31:44,628] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:31:44,629] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:31:44,679] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:31:44,683] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:44,683] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:44,992] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:44,993] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:31:45,071] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:31:45,072] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:31:45,101] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:31:45,102] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:31:45,128] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:31:45,210] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:31:45,237] - [task:165] INFO -  {'uptime': '241', 'memoryTotal': 15466930176, 'memoryFree': 13231800320, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:31:45,262] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:31:45,290] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:31:45,290] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:31:45,340] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:31:45,343] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:45,343] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:45,644] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:45,645] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:31:45,720] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:31:45,722] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:31:45,753] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:31:45,753] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:31:45,778] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:31:45,836] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:31:46,360] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:31:51,363] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:31:51,454] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:31:51,459] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:51,459] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:51,783] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:51,784] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:31:51,859] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:31:51,860] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:31:51,860] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:31:52,951] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:31:53,010] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 07:31:53,010] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:31:53,904] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:31:54,230] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:31:54,513] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:31:54,516] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:31:54,575] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:54,575] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:54,923] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:54,928] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:54,928] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:55,232] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:55,236] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:55,236] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:55,565] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:31:55,573] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:31:55,574] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:31:56,118] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:00,344] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:32:00,344] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.73759051628708, 'mem_free': 13223727104, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:32:00,344] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:32:00,344] - [basetestcase:467] INFO - Time to execute basesetup : 18.294050693511963
[2022-09-12 07:32:00,397] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:32:00,397] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:32:00,451] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:32:00,451] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:32:00,504] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:32:00,505] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:32:00,557] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:32:00,558] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:32:00,610] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:00,672] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:32:00,673] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:32:00,673] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:32:05,685] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:32:05,688] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:05,688] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:06,001] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:06,999] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:32:07,166] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:32:09,751] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:32:09,768] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 07:32:10,591] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:10,591] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:32:10,591] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:32:40,619] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:32:40,649] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:32:40,674] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:32:40,745] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.274611ms
[2022-09-12 07:32:40,745] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2cc65bbb-9830-4f08-aa54-b33e231b7804', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.274611ms', 'executionTime': '69.216297ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:32:40,745] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:32:40,772] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:32:40,799] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:32:41,514] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 713.037322ms
[2022-09-12 07:32:41,514] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:32:41,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:32:41,613] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:32:41,621] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.984532ms
[2022-09-12 07:32:41,828] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:41,865] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:32:41,883] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:32:41,883] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:32:41,900] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:32:41,976] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:42,789] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeefb1cc89f702343269bf53d12e5099da1job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:32:42,817] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeefb1cc89f702343269bf53d12e5099da1job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:32:42,866] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.769716ms
[2022-09-12 07:32:42,867] - [base_gsi:282] INFO - BUILD INDEX on default(employeefb1cc89f702343269bf53d12e5099da1job_title) USING GSI
[2022-09-12 07:32:43,898] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeefb1cc89f702343269bf53d12e5099da1job_title) USING GSI
[2022-09-12 07:32:43,926] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeefb1cc89f702343269bf53d12e5099da1job_title%29+USING+GSI
[2022-09-12 07:32:43,954] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.067299ms
[2022-09-12 07:32:44,985] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefb1cc89f702343269bf53d12e5099da1job_title'
[2022-09-12 07:32:45,012] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefb1cc89f702343269bf53d12e5099da1job_title%27
[2022-09-12 07:32:45,020] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.377898ms
[2022-09-12 07:32:46,049] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefb1cc89f702343269bf53d12e5099da1job_title'
[2022-09-12 07:32:46,075] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefb1cc89f702343269bf53d12e5099da1job_title%27
[2022-09-12 07:32:46,082] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.208366ms
[2022-09-12 07:32:46,542] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 07:32:46,707] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:32:47,411] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:32:48,085] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:48,118] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:32:48,145] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 07:32:48,149] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.276488ms
[2022-09-12 07:32:48,149] - [task:3246] INFO - {'requestID': '490401c2-7619-4d32-8a32-105764f60648', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeefb1cc89f702343269bf53d12e5099da1job_title', 'index_id': 'e928ff3456238b7d', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.276488ms', 'executionTime': '2.161239ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 07:32:48,150] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:48,150] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:48,151] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:32:48,151] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:32:48,151] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 07:32:48,152] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:32:48,152] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:32:48,152] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:32:48,152] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:32:49,152] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:49,182] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:32:49,209] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 07:32:49,311] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 93.000875ms
[2022-09-12 07:32:49,312] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:32:49,312] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:32:50,116] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:50,117] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:32:51,149] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefb1cc89f702343269bf53d12e5099da1job_title'
[2022-09-12 07:32:51,175] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefb1cc89f702343269bf53d12e5099da1job_title%27
[2022-09-12 07:32:51,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.780327ms
[2022-09-12 07:32:51,213] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeefb1cc89f702343269bf53d12e5099da1job_title ON default USING GSI
[2022-09-12 07:32:51,241] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeefb1cc89f702343269bf53d12e5099da1job_title+ON+default+USING+GSI
[2022-09-12 07:32:51,281] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.964835ms
[2022-09-12 07:32:51,319] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeefb1cc89f702343269bf53d12e5099da1job_title'
[2022-09-12 07:32:51,347] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeefb1cc89f702343269bf53d12e5099da1job_title%27
[2022-09-12 07:32:51,355] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.792686ms
[2022-09-12 07:32:51,355] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b24a22db-39ab-47f2-a119-8ecddcb9c3c5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.792686ms', 'executionTime': '5.715989ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:32:51,465] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:51,468] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:51,469] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:51,895] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:51,951] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:32:51,952] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:32:52,022] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:32:52,077] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:52,080] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:52,080] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:52,509] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:52,570] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:32:52,571] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:32:52,641] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:32:52,701] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:32:52,701] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:32:52,701] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:32:52,728] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:32:52,755] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:32:52,765] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.698038ms
[2022-09-12 07:32:52,792] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:32:52,819] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:32:52,874] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.159226ms
[2022-09-12 07:32:52,941] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:32:52,941] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 21.37924510358528, 'mem_free': 13020672000, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:32:52,941] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:32:52,946] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:52,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:53,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:53,399] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:53,399] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:54,091] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:54,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:54,101] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:54,856] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:32:54,866] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:32:54,866] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:32:55,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:00,106] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:33:00,253] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:33:01,514] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:33:01,543] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:33:01,544] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:33:01,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:33:01,663] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:33:01,719] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:33:01,720] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:33:01,808] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:33:01,809] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:33:01,838] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:01,988] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:33:01,988] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:02,018] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:33:02,046] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:33:02,047] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:02,075] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:33:02,103] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:33:02,104] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:02,132] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:33:02,161] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:33:02,161] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:02,190] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:33:02,190] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:33:02,190] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:33:02,191] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 80.196s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_4'}
[2022-09-12 07:33:02,294] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:02,295] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:02,723] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:02,758] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:33:02,841] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:33:02,841] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-12 07:33:02,842] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:33:03,321] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:33:03,347] - [task:165] INFO -  {'uptime': '324', 'memoryTotal': 15466930176, 'memoryFree': 13172269056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:33:03,373] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:33:03,373] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:33:03,374] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:33:03,405] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:33:03,447] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:33:03,448] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:33:03,475] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:33:03,476] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:33:03,476] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:33:03,477] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:33:03,525] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:33:03,528] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:03,528] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:03,904] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:03,905] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:33:03,999] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:33:04,000] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:33:04,032] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:33:04,033] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:33:04,059] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:33:04,148] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:33:04,177] - [task:165] INFO -  {'uptime': '320', 'memoryTotal': 15466930176, 'memoryFree': 13081845760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:33:04,204] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:33:04,234] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:33:04,234] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:33:04,286] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:33:04,289] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:04,289] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:04,697] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:04,699] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:33:04,788] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:33:04,789] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:33:04,820] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:33:04,821] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:33:04,848] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:33:04,934] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:33:04,966] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13073014784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:33:04,992] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:33:05,021] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:33:05,021] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:33:05,073] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:33:05,078] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:05,078] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:05,490] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:05,491] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:33:05,579] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:33:05,581] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:33:05,610] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:33:05,611] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:33:05,638] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:33:05,725] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:33:05,754] - [task:165] INFO -  {'uptime': '321', 'memoryTotal': 15466930176, 'memoryFree': 13176492032, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:33:05,781] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:33:05,812] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:33:05,813] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:33:05,866] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:33:05,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:05,869] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:06,255] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:06,256] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:33:06,340] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:33:06,341] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:33:06,371] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:33:06,372] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:33:06,397] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:33:06,454] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:33:06,993] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:33:11,999] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:33:12,083] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:33:12,087] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:12,088] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:12,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:12,484] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:33:12,580] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:33:12,580] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:33:12,581] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:33:13,580] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:33:13,642] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 07:33:13,642] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:33:14,553] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:33:14,881] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:33:15,152] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:33:15,155] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:33:15,208] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:15,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:15,609] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:15,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:15,614] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:15,987] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:15,991] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:15,992] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:16,593] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:16,601] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:16,602] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:17,292] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:21,885] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:33:21,886] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.52386262098567, 'mem_free': 13170659328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:33:21,886] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:33:21,886] - [basetestcase:467] INFO - Time to execute basesetup : 19.594261169433594
[2022-09-12 07:33:21,939] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:33:21,940] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:33:21,993] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:33:21,993] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:33:22,049] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:33:22,049] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:33:22,106] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:33:22,106] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:33:22,158] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:33:22,216] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:33:22,216] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:33:22,217] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:33:27,224] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:33:27,226] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:33:27,227] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:33:27,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:33:28,688] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:33:28,861] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:33:31,252] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:33:31,267] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 07:33:31,939] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:33:31,939] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:33:31,939] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:34:01,952] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:34:01,982] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:34:02,008] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:34:02,076] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.341796ms
[2022-09-12 07:34:02,076] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd1bc4eb8-e9ef-4836-b97b-9e80e7b1cbae', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.341796ms', 'executionTime': '66.278849ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:34:02,077] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:34:02,104] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:34:02,130] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:34:02,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 759.294924ms
[2022-09-12 07:34:02,892] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:34:02,941] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:34:02,991] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:34:02,999] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.838667ms
[2022-09-12 07:34:03,212] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:03,248] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:34:03,264] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:34:03,267] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:34:03,285] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:34:03,357] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:04,170] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeede858b490a734fe4948a5ae9873d4925job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:34:04,195] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeede858b490a734fe4948a5ae9873d4925job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:34:04,255] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.137845ms
[2022-09-12 07:34:04,255] - [base_gsi:282] INFO - BUILD INDEX on default(employeede858b490a734fe4948a5ae9873d4925job_title) USING GSI
[2022-09-12 07:34:05,286] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeede858b490a734fe4948a5ae9873d4925job_title) USING GSI
[2022-09-12 07:34:05,311] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeede858b490a734fe4948a5ae9873d4925job_title%29+USING+GSI
[2022-09-12 07:34:05,342] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.461286ms
[2022-09-12 07:34:06,373] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeede858b490a734fe4948a5ae9873d4925job_title'
[2022-09-12 07:34:06,398] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeede858b490a734fe4948a5ae9873d4925job_title%27
[2022-09-12 07:34:06,408] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.102032ms
[2022-09-12 07:34:07,441] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeede858b490a734fe4948a5ae9873d4925job_title'
[2022-09-12 07:34:07,467] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeede858b490a734fe4948a5ae9873d4925job_title%27
[2022-09-12 07:34:07,477] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.766193ms
[2022-09-12 07:34:07,952] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 07:34:08,124] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:34:08,960] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:34:09,142] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-12 07:34:09,143] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:34:09,309] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-12 07:34:09,309] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-12 07:34:09,309] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3180303870, 0, b'')
[2022-09-12 07:34:09,480] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:09,509] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:34:09,535] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 07:34:09,539] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.362996ms
[2022-09-12 07:34:09,540] - [task:3246] INFO - {'requestID': '38cd1f8e-99ac-4d30-a582-fefa67bf82a6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeede858b490a734fe4948a5ae9873d4925job_title', 'index_id': 'eebcac5c7cbbbab3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.362996ms', 'executionTime': '2.288974ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 07:34:09,540] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:09,540] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:09,540] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:34:09,541] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:34:09,541] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 07:34:09,541] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:34:09,542] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:34:09,542] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:34:09,542] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:34:10,541] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:10,571] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:34:10,597] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 07:34:10,716] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 111.101455ms
[2022-09-12 07:34:10,716] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:34:10,717] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:34:11,481] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:11,482] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:34:12,512] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeede858b490a734fe4948a5ae9873d4925job_title'
[2022-09-12 07:34:12,538] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeede858b490a734fe4948a5ae9873d4925job_title%27
[2022-09-12 07:34:12,548] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.251062ms
[2022-09-12 07:34:12,573] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeede858b490a734fe4948a5ae9873d4925job_title ON default USING GSI
[2022-09-12 07:34:12,597] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeede858b490a734fe4948a5ae9873d4925job_title+ON+default+USING+GSI
[2022-09-12 07:34:12,639] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.800042ms
[2022-09-12 07:34:12,671] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeede858b490a734fe4948a5ae9873d4925job_title'
[2022-09-12 07:34:12,699] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeede858b490a734fe4948a5ae9873d4925job_title%27
[2022-09-12 07:34:12,708] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.480926ms
[2022-09-12 07:34:12,708] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '4e51c914-bc4b-4363-a308-49e13f46039f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.480926ms', 'executionTime': '7.387594ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:34:12,820] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:12,823] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:12,823] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:13,283] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:13,340] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:34:13,340] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:34:13,417] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:34:13,473] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:13,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:13,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:13,956] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:14,018] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:34:14,018] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:34:14,115] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:34:14,182] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:14,183] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:34:14,183] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:34:14,209] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:34:14,236] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:34:14,248] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.664875ms
[2022-09-12 07:34:14,276] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:34:14,303] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:34:14,367] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.577267ms
[2022-09-12 07:34:14,439] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:34:14,439] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.11737587294279, 'mem_free': 13040197632, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:34:14,439] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:34:14,444] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:14,444] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:14,942] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:14,947] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:14,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:15,699] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:15,707] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:15,707] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:16,472] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:16,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:16,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:17,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:21,629] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:34:21,776] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:34:22,143] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:34:22,171] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:34:22,171] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:34:22,227] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:34:22,279] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:34:22,333] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:34:22,334] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:34:22,412] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:34:22,413] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:34:22,438] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:22,569] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:34:22,569] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:22,596] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:34:22,622] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:34:22,622] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:22,649] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:34:22,676] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:34:22,677] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:22,705] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:34:22,732] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:34:22,732] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:22,760] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:34:22,761] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:34:22,761] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:34:22,761] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 80.523s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_5'}
[2022-09-12 07:34:22,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:22,870] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:23,329] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:23,359] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:34:23,440] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:34:23,441] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-12 07:34:23,442] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:34:23,896] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:34:23,923] - [task:165] INFO -  {'uptime': '405', 'memoryTotal': 15466930176, 'memoryFree': 13128953856, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:34:23,951] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:34:23,951] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:34:23,952] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:34:23,981] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:34:24,023] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:34:24,023] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:34:24,052] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:34:24,053] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:34:24,053] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:34:24,053] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:34:24,101] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:34:24,104] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:24,105] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:24,507] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:24,508] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:34:24,595] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:34:24,596] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:34:24,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:34:24,628] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:34:24,654] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:34:24,741] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:34:24,770] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13037461504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:34:24,799] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:34:24,829] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:34:24,829] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:34:24,881] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:34:24,885] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:24,885] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:25,347] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:25,348] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:34:25,447] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:34:25,448] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:34:25,478] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:34:25,479] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:34:25,504] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:34:25,587] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:34:25,614] - [task:165] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13120475136, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:34:25,640] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:34:25,668] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:34:25,669] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:34:25,723] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:34:25,728] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:25,728] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:26,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:26,136] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:34:26,227] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:34:26,228] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:34:26,259] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:34:26,259] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:34:26,284] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:34:26,367] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:34:26,393] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13126131712, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:34:26,418] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:34:26,447] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:34:26,447] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:34:26,499] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:34:26,502] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:26,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:26,953] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:26,954] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:34:27,044] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:34:27,045] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:34:27,075] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:34:27,076] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:34:27,101] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:34:27,165] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:34:27,687] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:34:32,692] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:34:32,780] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:34:32,784] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:32,784] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:33,226] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:33,227] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:34:33,316] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:34:33,316] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:34:33,317] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:34:34,318] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:34:34,373] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:34:34,374] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:34:35,295] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:34:35,614] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:34:35,897] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:34:35,900] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:34:35,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:35,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:36,433] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:36,437] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:36,438] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:36,862] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:36,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:36,867] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:37,640] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:37,649] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:37,649] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:38,437] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:43,313] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:34:43,313] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.8067154990484, 'mem_free': 13130416128, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:34:43,313] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:34:43,314] - [basetestcase:467] INFO - Time to execute basesetup : 20.446422815322876
[2022-09-12 07:34:43,369] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:34:43,369] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:34:43,425] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:34:43,426] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:34:43,483] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:34:43,484] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:34:43,541] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:34:43,541] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:34:43,592] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:43,656] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:34:43,657] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:34:43,658] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:34:48,669] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:34:48,673] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:34:48,674] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:34:49,085] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:34:50,138] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:34:50,308] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:34:52,674] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:34:52,691] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 07:34:53,747] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:34:53,748] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:34:53,748] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:35:23,759] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:35:23,789] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:35:23,817] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:35:23,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.942084ms
[2022-09-12 07:35:23,890] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7416b448-5aaa-4506-b048-b40b5acc8dab', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.942084ms', 'executionTime': '70.877531ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:35:23,890] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:35:23,918] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:35:23,944] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:35:24,618] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 672.324681ms
[2022-09-12 07:35:24,619] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:35:24,691] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:35:24,722] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:35:24,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.020982ms
[2022-09-12 07:35:24,935] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:35:24,969] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:35:24,983] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:35:24,984] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:35:24,995] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:35:25,091] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:35:25,898] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeebcb7e23daaef48f1830ef993f58a2f28job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:35:25,924] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeebcb7e23daaef48f1830ef993f58a2f28job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:35:25,985] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 58.729491ms
[2022-09-12 07:35:25,986] - [base_gsi:282] INFO - BUILD INDEX on default(employeebcb7e23daaef48f1830ef993f58a2f28job_title) USING GSI
[2022-09-12 07:35:27,016] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeebcb7e23daaef48f1830ef993f58a2f28job_title) USING GSI
[2022-09-12 07:35:27,042] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeebcb7e23daaef48f1830ef993f58a2f28job_title%29+USING+GSI
[2022-09-12 07:35:27,067] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.550916ms
[2022-09-12 07:35:28,099] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebcb7e23daaef48f1830ef993f58a2f28job_title'
[2022-09-12 07:35:28,127] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebcb7e23daaef48f1830ef993f58a2f28job_title%27
[2022-09-12 07:35:28,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.005404ms
[2022-09-12 07:35:29,169] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebcb7e23daaef48f1830ef993f58a2f28job_title'
[2022-09-12 07:35:29,194] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebcb7e23daaef48f1830ef993f58a2f28job_title%27
[2022-09-12 07:35:29,205] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.507136ms
[2022-09-12 07:35:29,683] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:35:29,858] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:35:32,652] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:35:33,209] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:33,241] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:35:33,268] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 07:35:33,273] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.7168ms
[2022-09-12 07:35:33,273] - [task:3246] INFO - {'requestID': 'fe33c5f9-490e-427a-a245-55085f9b77d8', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeebcb7e23daaef48f1830ef993f58a2f28job_title', 'index_id': 'ad23ed02fc405468', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.7168ms', 'executionTime': '2.642485ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 07:35:33,273] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:33,274] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:33,274] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:35:33,274] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:35:33,275] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 07:35:33,275] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:35:33,275] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:35:33,275] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:35:33,276] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:35:34,275] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:34,305] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:35:34,329] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 07:35:34,514] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 171.597307ms
[2022-09-12 07:35:34,515] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:35:34,516] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:35:36,048] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:36,048] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:35:37,079] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebcb7e23daaef48f1830ef993f58a2f28job_title'
[2022-09-12 07:35:37,104] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebcb7e23daaef48f1830ef993f58a2f28job_title%27
[2022-09-12 07:35:37,114] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.357023ms
[2022-09-12 07:35:37,142] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeebcb7e23daaef48f1830ef993f58a2f28job_title ON default USING GSI
[2022-09-12 07:35:37,167] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeebcb7e23daaef48f1830ef993f58a2f28job_title+ON+default+USING+GSI
[2022-09-12 07:35:37,226] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.632227ms
[2022-09-12 07:35:37,263] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeebcb7e23daaef48f1830ef993f58a2f28job_title'
[2022-09-12 07:35:37,289] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeebcb7e23daaef48f1830ef993f58a2f28job_title%27
[2022-09-12 07:35:37,297] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.695732ms
[2022-09-12 07:35:37,297] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a9494d14-a09f-4742-baf9-521bbdaebd68', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.695732ms', 'executionTime': '5.638396ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:35:37,398] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:35:37,401] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:37,401] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:37,906] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:37,961] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:35:37,961] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:35:38,050] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:35:38,105] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:35:38,108] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:38,108] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:38,622] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:38,679] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:35:38,679] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:35:38,767] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:35:38,822] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:35:38,823] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:35:38,823] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:35:38,848] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:35:38,873] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:35:38,883] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.195786ms
[2022-09-12 07:35:38,908] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:35:38,933] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:35:38,986] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 50.809582ms
[2022-09-12 07:35:39,049] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:35:39,049] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 14.1478613435475, 'mem_free': 12970860544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:35:39,049] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:35:39,054] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:39,054] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:39,564] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:39,571] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:39,571] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:40,138] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:40,143] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:40,143] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:40,836] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:40,844] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:40,844] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:41,777] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:48,182] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:35:48,330] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:35:48,884] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:35:48,911] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:35:48,911] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:35:48,965] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:35:49,019] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:35:49,071] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:35:49,072] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:35:49,151] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:35:49,152] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:35:49,180] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:49,310] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:35:49,310] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:49,336] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:35:49,361] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:35:49,361] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:49,387] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:35:49,412] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:35:49,412] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:49,440] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:35:49,476] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:35:49,476] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:49,504] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:35:49,504] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:35:49,504] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:35:49,505] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 86.689s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_6'}
[2022-09-12 07:35:49,611] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:49,611] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:50,111] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:50,150] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:35:50,228] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:35:50,228] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-12 07:35:50,229] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:35:50,639] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:35:50,668] - [task:165] INFO -  {'uptime': '492', 'memoryTotal': 15466930176, 'memoryFree': 13059796992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:35:50,699] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:35:50,699] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:35:50,700] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:35:50,731] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:35:50,768] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:35:50,769] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:35:50,798] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:35:50,799] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:35:50,799] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:35:50,800] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:35:50,849] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:35:50,852] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:50,852] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:51,408] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:51,409] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:35:51,519] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:35:51,520] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:35:51,551] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:35:51,552] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:35:51,577] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:35:51,663] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:35:51,689] - [task:165] INFO -  {'uptime': '491', 'memoryTotal': 15466930176, 'memoryFree': 13059538944, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:35:51,716] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:35:51,743] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:35:51,744] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:35:51,794] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:35:51,798] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:51,798] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:52,280] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:52,281] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:35:52,380] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:35:52,381] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:35:52,410] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:35:52,411] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:35:52,435] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:35:52,518] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:35:52,544] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13059661824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:35:52,569] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:35:52,596] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:35:52,597] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:35:52,647] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:35:52,652] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:52,652] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:53,183] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:53,185] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:35:53,298] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:35:53,299] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:35:53,330] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:35:53,331] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:35:53,357] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:35:53,450] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:35:53,479] - [task:165] INFO -  {'uptime': '487', 'memoryTotal': 15466930176, 'memoryFree': 13059166208, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:35:53,506] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:35:53,535] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:35:53,535] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:35:53,585] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:35:53,588] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:53,588] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:35:54,121] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:35:54,122] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:35:54,237] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:35:54,239] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:35:54,270] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:35:54,270] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:35:54,298] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:35:54,357] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:35:54,895] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:35:59,898] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:35:59,988] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:35:59,993] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:35:59,993] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:00,563] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:00,564] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:36:00,670] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:36:00,671] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:36:00,671] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:36:01,497] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:36:01,552] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:36:01,553] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:36:02,593] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:36:02,922] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:36:03,205] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:36:03,208] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:36:03,266] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:36:03,266] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:03,810] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:03,815] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:36:03,815] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:04,437] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:04,446] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:36:04,446] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:05,358] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:05,366] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:36:05,366] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:06,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:12,049] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:36:12,049] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.34788698330848, 'mem_free': 13066006528, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:36:12,050] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:36:12,050] - [basetestcase:467] INFO - Time to execute basesetup : 22.440953016281128
[2022-09-12 07:36:12,101] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:36:12,102] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:36:12,156] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:36:12,156] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:36:12,210] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:36:12,210] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:36:12,267] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:36:12,267] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:36:12,322] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:36:12,382] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:36:12,382] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:36:12,383] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:36:17,394] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:36:17,398] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:36:17,399] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:36:17,893] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:36:18,985] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:36:19,161] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:36:21,449] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:36:21,468] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 07:36:22,964] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:36:22,964] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:36:22,964] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:36:52,992] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:36:53,025] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:36:53,052] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:36:53,121] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.138696ms
[2022-09-12 07:36:53,122] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'af00a48f-3ea6-4a6a-8afd-e8f64c740c63', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.138696ms', 'executionTime': '67.080154ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:36:53,122] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:36:53,148] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:36:53,175] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:36:53,882] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 705.329654ms
[2022-09-12 07:36:53,883] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:36:53,956] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:36:53,988] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:36:53,996] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.718994ms
[2022-09-12 07:36:54,210] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:36:54,249] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:36:54,267] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:36:54,267] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:36:54,282] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:36:54,367] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:36:55,171] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee2ba8755bbd44bdf9aeb6073878f3248job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:36:55,197] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee2ba8755bbd44bdf9aeb6073878f3248job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:36:55,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.415584ms
[2022-09-12 07:36:55,266] - [base_gsi:282] INFO - BUILD INDEX on default(employeee2ba8755bbd44bdf9aeb6073878f3248job_title) USING GSI
[2022-09-12 07:36:56,295] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee2ba8755bbd44bdf9aeb6073878f3248job_title) USING GSI
[2022-09-12 07:36:56,323] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee2ba8755bbd44bdf9aeb6073878f3248job_title%29+USING+GSI
[2022-09-12 07:36:56,348] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.152436ms
[2022-09-12 07:36:57,380] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee2ba8755bbd44bdf9aeb6073878f3248job_title'
[2022-09-12 07:36:57,406] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee2ba8755bbd44bdf9aeb6073878f3248job_title%27
[2022-09-12 07:36:57,417] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.785763ms
[2022-09-12 07:36:58,448] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee2ba8755bbd44bdf9aeb6073878f3248job_title'
[2022-09-12 07:36:58,476] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee2ba8755bbd44bdf9aeb6073878f3248job_title%27
[2022-09-12 07:36:58,485] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.435778ms
[2022-09-12 07:36:58,864] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 07:36:59,036] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:37:00,117] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:37:00,748] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 07:37:00,930] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:37:02,251] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:37:02,879] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:37:03,056] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:37:06,566] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:37:07,506] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:07,538] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:37:07,564] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 07:37:07,568] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.896323ms
[2022-09-12 07:37:07,569] - [task:3246] INFO - {'requestID': 'eb87612e-323c-4c49-94d3-0472c58ece39', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee2ba8755bbd44bdf9aeb6073878f3248job_title', 'index_id': '63cdb3ac2b262b24', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.896323ms', 'executionTime': '2.838054ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 07:37:07,569] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:07,569] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:07,570] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:37:07,570] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:37:07,570] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 07:37:07,571] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:37:07,571] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:37:07,571] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:37:07,571] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 07:37:08,571] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:08,601] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 07:37:08,627] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 07:37:08,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 170.920959ms
[2022-09-12 07:37:08,814] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:37:08,815] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:37:10,348] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:10,348] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 07:37:11,379] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee2ba8755bbd44bdf9aeb6073878f3248job_title'
[2022-09-12 07:37:11,404] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee2ba8755bbd44bdf9aeb6073878f3248job_title%27
[2022-09-12 07:37:11,414] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.372657ms
[2022-09-12 07:37:11,440] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee2ba8755bbd44bdf9aeb6073878f3248job_title ON default USING GSI
[2022-09-12 07:37:11,465] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee2ba8755bbd44bdf9aeb6073878f3248job_title+ON+default+USING+GSI
[2022-09-12 07:37:11,514] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.423571ms
[2022-09-12 07:37:11,547] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee2ba8755bbd44bdf9aeb6073878f3248job_title'
[2022-09-12 07:37:11,577] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee2ba8755bbd44bdf9aeb6073878f3248job_title%27
[2022-09-12 07:37:11,585] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.026621ms
[2022-09-12 07:37:11,586] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a28d6e23-dfa0-43bd-a40b-9621439fdac6', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.026621ms', 'executionTime': '5.943061ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:37:11,696] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:37:11,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:11,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:12,360] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:12,416] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:37:12,416] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:37:12,519] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:37:12,575] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:37:12,578] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:12,578] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:13,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:13,283] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:37:13,283] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:37:13,389] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:37:13,446] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:37:13,446] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:37:13,446] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:37:13,471] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:37:13,496] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:37:13,506] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.379456ms
[2022-09-12 07:37:13,536] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:37:13,564] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:37:13,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.476917ms
[2022-09-12 07:37:13,691] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:37:13,692] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.56966670156975, 'mem_free': 12871241728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:37:13,692] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:37:13,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:13,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:14,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:14,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:14,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:15,026] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:15,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:15,032] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:15,898] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:15,905] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:15,906] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:17,063] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:24,752] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:37:24,948] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:37:26,084] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:37:26,115] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:37:26,115] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:37:26,174] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:37:26,231] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:37:26,282] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:37:26,283] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:37:26,361] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:37:26,362] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:37:26,387] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:26,516] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:37:26,516] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:26,543] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:37:26,568] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:37:26,569] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:26,596] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:37:26,622] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:37:26,622] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:26,649] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:37:26,676] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:37:26,676] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:26,706] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:37:26,706] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 07:37:26,706] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:37:26,707] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.152s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_7'}
[2022-09-12 07:37:26,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:26,848] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:27,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:27,495] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:37:27,577] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:37:27,577] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-12 07:37:27,578] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:37:27,875] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:37:27,904] - [task:165] INFO -  {'uptime': '589', 'memoryTotal': 15466930176, 'memoryFree': 12995141632, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:37:27,930] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:37:27,930] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:37:27,931] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:37:27,969] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:37:28,003] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:37:28,004] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:37:28,034] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:37:28,035] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:37:28,035] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:37:28,036] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:37:28,085] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:37:28,089] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:28,089] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:28,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:28,695] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:37:28,807] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:37:28,808] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:37:28,839] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:37:28,840] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:37:28,867] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:37:28,954] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:37:28,982] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12996026368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:37:29,009] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:37:29,037] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:37:29,037] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:37:29,088] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:37:29,091] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:29,092] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:29,738] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:29,740] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:37:29,864] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:37:29,865] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:37:29,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:37:29,896] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:37:29,923] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:37:30,009] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:37:30,039] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12995674112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:37:30,068] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:37:30,098] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:37:30,098] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:37:30,152] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:37:30,157] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:30,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:30,796] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:30,797] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:37:30,922] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:37:30,923] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:37:30,952] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:37:30,953] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:37:30,978] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:37:31,063] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:37:31,091] - [task:165] INFO -  {'uptime': '587', 'memoryTotal': 15466930176, 'memoryFree': 12995366912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:37:31,124] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:37:31,152] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:37:31,153] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:37:31,206] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:37:31,209] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:31,209] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:31,848] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:31,849] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:37:31,978] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:37:31,980] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:37:32,011] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:37:32,011] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:37:32,037] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:37:32,096] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:37:32,607] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:37:37,612] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:37:37,696] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:37:37,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:37,701] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:38,306] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:38,307] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:37:38,422] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:37:38,423] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:37:38,423] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:37:39,232] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:37:39,285] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:37:39,286] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:37:40,180] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:37:40,489] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:37:40,777] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:37:40,780] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 07:37:40,838] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:40,838] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:41,461] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:41,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:41,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:42,350] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:42,359] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:42,359] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:43,431] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:43,441] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:43,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:44,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:51,008] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:37:51,008] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 32.35380439467298, 'mem_free': 13010604032, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:37:51,008] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:37:51,009] - [basetestcase:467] INFO - Time to execute basesetup : 24.16362977027893
[2022-09-12 07:37:51,062] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:37:51,063] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:37:51,120] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:37:51,120] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:37:51,177] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:37:51,177] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:37:51,242] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:37:51,242] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:37:51,293] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:37:51,357] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:37:51,359] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:37:51,359] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:37:56,371] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:37:56,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:37:56,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:37:56,998] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:37:58,034] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:37:58,306] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:38:00,472] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:38:00,555] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:38:00,555] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:38:00,556] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:38:30,576] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:38:30,606] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:38:30,634] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:38:30,706] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.640693ms
[2022-09-12 07:38:30,706] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '546f9bac-4d2a-4508-b515-2f955a837f53', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.640693ms', 'executionTime': '69.581617ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:38:30,706] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:38:30,734] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:38:30,759] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:38:31,426] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 664.287398ms
[2022-09-12 07:38:31,426] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:38:31,478] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:38:31,513] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:38:31,522] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.533229ms
[2022-09-12 07:38:31,729] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:38:31,767] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:38:31,784] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:38:31,784] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:38:31,803] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:38:31,806] - [base_gsi:326] INFO - []
[2022-09-12 07:38:32,697] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5949ac213987415f90c2a4ab40cff9d0job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:38:32,722] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5949ac213987415f90c2a4ab40cff9d0job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:38:32,786] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.442028ms
[2022-09-12 07:38:32,819] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee5949ac213987415f90c2a4ab40cff9d0join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:38:32,846] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee5949ac213987415f90c2a4ab40cff9d0join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:38:32,901] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.184014ms
[2022-09-12 07:38:32,902] - [base_gsi:282] INFO - BUILD INDEX on default(employee5949ac213987415f90c2a4ab40cff9d0job_title,employee5949ac213987415f90c2a4ab40cff9d0join_yr) USING GSI
[2022-09-12 07:38:33,932] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee5949ac213987415f90c2a4ab40cff9d0job_title,employee5949ac213987415f90c2a4ab40cff9d0join_yr) USING GSI
[2022-09-12 07:38:33,959] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee5949ac213987415f90c2a4ab40cff9d0job_title%2Cemployee5949ac213987415f90c2a4ab40cff9d0join_yr%29+USING+GSI
[2022-09-12 07:38:34,005] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.779785ms
[2022-09-12 07:38:35,036] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0job_title'
[2022-09-12 07:38:35,061] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0job_title%27
[2022-09-12 07:38:35,069] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.236364ms
[2022-09-12 07:38:36,100] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0job_title'
[2022-09-12 07:38:36,128] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0job_title%27
[2022-09-12 07:38:36,136] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.096709ms
[2022-09-12 07:38:36,165] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0join_yr'
[2022-09-12 07:38:36,192] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0join_yr%27
[2022-09-12 07:38:36,195] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.695346ms
[2022-09-12 07:38:37,227] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0job_title'
[2022-09-12 07:38:37,252] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0job_title%27
[2022-09-12 07:38:37,262] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.150629ms
[2022-09-12 07:38:37,287] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5949ac213987415f90c2a4ab40cff9d0job_title ON default USING GSI
[2022-09-12 07:38:37,312] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5949ac213987415f90c2a4ab40cff9d0job_title+ON+default+USING+GSI
[2022-09-12 07:38:37,356] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.160207ms
[2022-09-12 07:38:37,389] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0join_yr'
[2022-09-12 07:38:37,414] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0join_yr%27
[2022-09-12 07:38:37,422] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.88481ms
[2022-09-12 07:38:37,448] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee5949ac213987415f90c2a4ab40cff9d0join_yr ON default USING GSI
[2022-09-12 07:38:37,473] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee5949ac213987415f90c2a4ab40cff9d0join_yr+ON+default+USING+GSI
[2022-09-12 07:38:37,510] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.023286ms
[2022-09-12 07:38:37,545] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0job_title'
[2022-09-12 07:38:37,570] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0job_title%27
[2022-09-12 07:38:37,578] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.629502ms
[2022-09-12 07:38:37,578] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e4d558ed-e9ce-45b2-ba11-f722494eb0be', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.629502ms', 'executionTime': '5.567005ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:38:37,604] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee5949ac213987415f90c2a4ab40cff9d0join_yr'
[2022-09-12 07:38:37,631] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee5949ac213987415f90c2a4ab40cff9d0join_yr%27
[2022-09-12 07:38:37,633] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.196684ms
[2022-09-12 07:38:37,634] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '57548e77-eb01-47dc-a8a1-c01a21ddd3e2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.196684ms', 'executionTime': '1.132021ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:38:37,732] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:38:37,735] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:37,735] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:38,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:38,384] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:38:38,384] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:38:38,481] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:38:38,536] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:38:38,539] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:38,539] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:39,132] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:39,189] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:38:39,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:38:39,294] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:38:39,350] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:38:39,350] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:38:39,351] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:38:39,376] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:38:39,401] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:38:39,410] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.530509ms
[2022-09-12 07:38:39,436] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:38:39,461] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:38:39,520] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.879034ms
[2022-09-12 07:38:39,583] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:38:39,584] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.581221755064415, 'mem_free': 12890136576, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:38:39,584] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:38:39,588] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:39,588] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:40,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:40,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:40,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:40,851] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:40,857] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:40,857] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:41,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:41,845] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:41,845] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:42,911] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:50,196] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-12 07:38:50,341] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:38:50,761] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:38:50,790] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:38:50,790] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:38:50,848] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:38:50,909] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:38:50,970] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:38:50,971] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:38:51,059] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:38:51,060] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:38:51,087] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:51,225] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:38:51,225] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:51,254] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:38:51,282] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:38:51,282] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:51,310] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:38:51,337] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:38:51,337] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:51,366] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:38:51,395] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:38:51,396] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:51,425] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:38:51,426] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 07:38:51,426] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:38:51,427] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.665s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_8'}
[2022-09-12 07:38:51,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:51,548] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:52,153] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:52,185] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:38:52,265] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:38:52,265] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-12 07:38:52,265] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:38:52,573] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:38:52,599] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 13019058176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:38:52,625] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:38:52,625] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:38:52,625] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:38:52,658] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:38:52,694] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:38:52,695] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:38:52,724] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:38:52,725] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:38:52,725] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:38:52,726] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:38:52,778] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:38:52,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:52,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:53,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:53,360] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:38:53,471] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:38:53,472] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:38:53,503] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:38:53,504] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:38:53,529] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:38:53,612] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:38:53,638] - [task:165] INFO -  {'uptime': '672', 'memoryTotal': 15466930176, 'memoryFree': 13019942912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:38:53,664] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:38:53,691] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:38:53,691] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:38:53,741] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:38:53,744] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:53,744] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:54,386] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:54,387] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:38:54,512] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:38:54,513] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:38:54,544] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:38:54,545] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:38:54,571] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:38:54,661] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:38:54,689] - [task:165] INFO -  {'uptime': '673', 'memoryTotal': 15466930176, 'memoryFree': 13012836352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:38:54,717] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:38:54,746] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:38:54,746] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:38:54,799] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:38:54,804] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:54,804] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:55,447] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:55,448] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:38:55,571] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:38:55,572] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:38:55,601] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:38:55,602] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:38:55,628] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:38:55,711] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:38:55,741] - [task:165] INFO -  {'uptime': '668', 'memoryTotal': 15466930176, 'memoryFree': 13019971584, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:38:55,769] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:38:55,801] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:38:55,801] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:38:55,851] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:38:55,855] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:38:55,855] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:38:56,484] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:38:56,485] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:38:56,608] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:38:56,609] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:38:56,638] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:38:56,639] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:38:56,666] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:38:56,726] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:38:57,255] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:39:02,261] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:39:02,350] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:39:02,355] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:02,355] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:03,018] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:03,019] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:39:03,143] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:39:03,144] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:39:03,145] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:39:03,843] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:39:03,897] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:39:03,898] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:39:04,831] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:39:05,150] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:39:05,446] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:39:05,449] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 07:39:05,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:05,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:06,124] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:06,129] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:06,129] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:06,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:06,993] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:06,993] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:08,043] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:08,053] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:08,053] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:09,154] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:15,825] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:39:15,826] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.33070174350785, 'mem_free': 13020823552, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:39:15,826] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:39:15,826] - [basetestcase:467] INFO - Time to execute basesetup : 24.28116774559021
[2022-09-12 07:39:15,882] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:39:15,882] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:39:15,939] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:39:15,939] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:39:15,994] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:39:15,995] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:39:16,056] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:39:16,056] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:39:16,111] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:39:16,179] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:39:16,181] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:39:16,181] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:39:21,194] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:39:21,198] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:39:21,198] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:39:21,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:39:22,851] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:39:23,146] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:39:25,249] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:39:25,328] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:39:25,328] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:39:25,329] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:39:55,358] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:39:55,388] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:39:55,414] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:39:55,487] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.641395ms
[2022-09-12 07:39:55,487] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'cf95db04-393b-4e70-9e9a-77db6b5dea38', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.641395ms', 'executionTime': '70.582147ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:39:55,487] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:39:55,517] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:39:55,543] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:39:56,273] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 728.387145ms
[2022-09-12 07:39:56,274] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:39:56,339] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:39:56,376] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:39:56,384] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.973343ms
[2022-09-12 07:39:56,578] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:39:56,616] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:39:56,642] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:39:56,643] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:39:56,673] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:39:56,673] - [base_gsi:326] INFO - []
[2022-09-12 07:39:57,545] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:39:57,572] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:39:57,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.445707ms
[2022-09-12 07:39:57,628] - [base_gsi:282] INFO - BUILD INDEX on default(employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr) USING GSI
[2022-09-12 07:39:58,658] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr) USING GSI
[2022-09-12 07:39:58,683] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%29+USING+GSI
[2022-09-12 07:39:58,710] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.484962ms
[2022-09-12 07:39:59,744] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr'
[2022-09-12 07:39:59,772] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%27
[2022-09-12 07:39:59,781] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.957833ms
[2022-09-12 07:40:00,812] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr'
[2022-09-12 07:40:00,842] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%27
[2022-09-12 07:40:00,851] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.669541ms
[2022-09-12 07:40:01,883] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr'
[2022-09-12 07:40:01,913] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%27
[2022-09-12 07:40:01,922] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.006189ms
[2022-09-12 07:40:01,952] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr ON default USING GSI
[2022-09-12 07:40:01,981] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr+ON+default+USING+GSI
[2022-09-12 07:40:02,015] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.780012ms
[2022-09-12 07:40:02,057] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr'
[2022-09-12 07:40:02,087] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3a4beda4ba334440bf804a9ad978f12ejob_title_join_yr%27
[2022-09-12 07:40:02,094] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.794797ms
[2022-09-12 07:40:02,094] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '739e63ef-86e7-4f47-a046-978de96cd759', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.794797ms', 'executionTime': '5.734098ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:40:02,209] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:40:02,212] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:02,212] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:02,892] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:02,949] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:40:02,950] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:40:03,059] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:40:03,119] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:40:03,122] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:03,122] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:03,783] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:03,839] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:40:03,839] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:40:03,963] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:40:04,023] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:40:04,023] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:40:04,024] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:40:04,051] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:40:04,079] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:40:04,089] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.517091ms
[2022-09-12 07:40:04,119] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:40:04,146] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:40:04,199] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.304277ms
[2022-09-12 07:40:04,272] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:40:04,273] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.56468370632587, 'mem_free': 12869087232, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:40:04,273] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:40:04,277] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:04,277] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:04,955] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:04,960] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:04,961] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:05,617] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:05,622] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:05,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:06,494] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:06,501] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:06,502] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:07,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:15,107] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-12 07:40:15,258] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:40:16,434] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:40:16,462] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:40:16,463] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:40:16,522] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:40:16,574] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:40:16,626] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:40:16,627] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:40:16,706] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:40:16,707] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:40:16,732] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:16,864] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:40:16,865] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:16,892] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:40:16,919] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:40:16,919] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:16,945] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:40:16,970] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:40:16,970] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:16,997] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:40:17,023] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:40:17,024] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:17,050] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:40:17,050] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 07:40:17,050] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:40:17,051] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 85.560s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_9'}
[2022-09-12 07:40:17,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:17,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:17,812] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:17,846] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:40:17,928] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:40:17,928] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-12 07:40:17,929] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:40:18,176] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:40:18,206] - [task:165] INFO -  {'uptime': '759', 'memoryTotal': 15466930176, 'memoryFree': 13011136512, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:40:18,235] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:40:18,235] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:40:18,236] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:40:18,275] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:40:18,309] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:40:18,309] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:40:18,339] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:40:18,340] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:40:18,341] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:40:18,341] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:40:18,391] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:40:18,394] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:18,394] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:19,047] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:19,048] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:40:19,167] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:40:19,168] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:40:19,198] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:40:19,199] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:40:19,226] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:40:19,315] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:40:19,343] - [task:165] INFO -  {'uptime': '757', 'memoryTotal': 15466930176, 'memoryFree': 13009764352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:40:19,369] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:40:19,396] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:40:19,396] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:40:19,449] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:40:19,452] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:19,452] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:20,094] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:20,095] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:40:20,220] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:40:20,221] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:40:20,251] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:40:20,252] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:40:20,279] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:40:20,371] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:40:20,400] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 13008826368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:40:20,427] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:40:20,457] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:40:20,457] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:40:20,509] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:40:20,514] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:20,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:21,180] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:21,182] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:40:21,311] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:40:21,312] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:40:21,344] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:40:21,345] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:40:21,375] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:40:21,471] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:40:21,499] - [task:165] INFO -  {'uptime': '758', 'memoryTotal': 15466930176, 'memoryFree': 13009027072, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:40:21,527] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:40:21,555] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:40:21,555] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:40:21,610] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:40:21,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:21,613] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:22,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:22,239] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:40:22,357] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:40:22,358] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:40:22,388] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:40:22,389] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:40:22,414] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:40:22,469] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:40:23,007] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:40:28,007] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:40:28,097] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:40:28,102] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:28,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:28,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:28,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:40:28,897] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:40:28,899] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:40:28,899] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:40:29,585] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:40:29,640] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:40:29,641] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:40:30,591] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:40:30,925] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:40:31,238] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:40:31,240] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 07:40:31,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:31,308] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:31,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:31,946] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:31,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:32,813] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:32,825] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:32,826] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:33,962] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:33,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:33,976] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:35,145] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:41,740] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:40:41,741] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.66081285248388, 'mem_free': 13023891456, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:40:41,741] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:40:41,741] - [basetestcase:467] INFO - Time to execute basesetup : 24.595734357833862
[2022-09-12 07:40:41,792] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:40:41,792] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:40:41,846] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:40:41,847] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:40:41,903] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:40:41,903] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:40:41,961] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:40:41,961] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:40:42,015] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:40:42,081] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:40:42,081] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:40:42,082] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:40:47,090] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:40:47,094] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:40:47,094] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:40:47,709] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:40:48,744] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:40:49,026] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:40:51,137] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:40:51,216] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:40:51,217] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:40:51,217] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:41:21,226] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:41:21,255] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:41:21,279] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:41:21,347] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.888656ms
[2022-09-12 07:41:21,347] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0c986ee7-07f1-491e-b7a9-862e44cb674f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.888656ms', 'executionTime': '65.810367ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:41:21,347] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:41:21,373] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:41:21,398] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:41:22,145] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 744.800439ms
[2022-09-12 07:41:22,145] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:41:22,226] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:41:22,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:41:22,269] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.841617ms
[2022-09-12 07:41:22,451] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:22,493] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:41:22,510] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:41:22,511] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:41:22,527] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:41:22,607] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:23,416] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6b50fdfd943343ec97a68ad2636f619fjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:41:23,441] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:41:23,500] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.413464ms
[2022-09-12 07:41:23,503] - [base_gsi:282] INFO - BUILD INDEX on default(`employee6b50fdfd943343ec97a68ad2636f619fjoin_yr`) USING GSI
[2022-09-12 07:41:24,534] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee6b50fdfd943343ec97a68ad2636f619fjoin_yr`) USING GSI
[2022-09-12 07:41:24,559] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%60%29+USING+GSI
[2022-09-12 07:41:24,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.289625ms
[2022-09-12 07:41:24,633] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b50fdfd943343ec97a68ad2636f619fjoin_yr'
[2022-09-12 07:41:24,665] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%27
[2022-09-12 07:41:24,674] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.24006ms
[2022-09-12 07:41:25,704] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b50fdfd943343ec97a68ad2636f619fjoin_yr'
[2022-09-12 07:41:25,730] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%27
[2022-09-12 07:41:25,738] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.328546ms
[2022-09-12 07:41:26,769] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b50fdfd943343ec97a68ad2636f619fjoin_yr'
[2022-09-12 07:41:26,796] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%27
[2022-09-12 07:41:26,803] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.835286ms
[2022-09-12 07:41:27,834] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:41:27,859] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 07:41:27,864] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.015905ms
[2022-09-12 07:41:27,864] - [base_gsi:504] INFO - {'requestID': '68d74be9-1fd5-4ba2-8cfd-9ca02dcfd9c9', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6b50fdfd943343ec97a68ad2636f619fjoin_yr', 'index_id': 'eb1a5618d98f5a74', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.015905ms', 'executionTime': '2.949449ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 07:41:27,865] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:41:27,865] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:41:27,865] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:41:27,865] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:41:27,866] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:41:27,867] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:41:27,867] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:41:27,867] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:41:27,927] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:41:27,967] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:41:27,992] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 07:41:28,178] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.050461ms
[2022-09-12 07:41:28,178] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:41:28,179] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:41:30,175] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:30,202] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6b50fdfd943343ec97a68ad2636f619fjoin_yr'
[2022-09-12 07:41:30,228] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6b50fdfd943343ec97a68ad2636f619fjoin_yr%27
[2022-09-12 07:41:30,234] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.393512ms
[2022-09-12 07:41:30,235] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '93335682-b09f-4a72-89d5-4b13b40dc4e0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.393512ms', 'executionTime': '4.321981ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:41:30,344] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:30,347] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:30,347] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:31,127] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:31,186] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:41:31,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:41:31,295] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:41:31,351] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:31,354] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:31,354] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:32,035] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:32,090] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:41:32,090] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:41:32,224] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:41:32,280] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:41:32,281] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:41:32,281] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:41:32,307] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:41:32,333] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:41:32,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.281912ms
[2022-09-12 07:41:32,339] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6a7a4ade-37f1-457e-a0ee-2c034cbbf686', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.281912ms', 'executionTime': '3.222002ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:41:32,391] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:41:32,392] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.07080897397691, 'mem_free': 12865351680, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:41:32,392] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:41:32,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:32,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:33,095] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:33,099] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:33,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:33,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:33,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:33,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:34,840] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:34,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:34,848] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:36,050] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:43,686] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-12 07:41:43,823] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:41:43,878] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:41:43,934] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:41:43,996] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:41:43,996] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:41:44,079] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:41:44,080] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:41:44,108] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:44,246] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:41:44,247] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:44,275] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:41:44,303] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:41:44,304] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:44,332] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:41:44,361] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:41:44,361] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:44,390] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:41:44,417] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:41:44,418] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:44,446] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:41:44,447] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 07:41:44,447] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:41:44,448] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 87.355s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_10'}
[2022-09-12 07:41:44,534] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:44,534] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:45,236] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:45,269] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:41:45,356] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:41:45,356] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-12 07:41:45,357] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:41:45,561] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:41:45,588] - [task:165] INFO -  {'uptime': '848', 'memoryTotal': 15466930176, 'memoryFree': 13013020672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:41:45,615] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:41:45,615] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:41:45,616] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:41:45,649] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:41:45,686] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:41:45,686] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:41:45,716] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:41:45,717] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:41:45,717] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:41:45,717] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:41:45,766] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:41:45,769] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:45,769] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:46,447] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:46,448] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:41:46,577] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:41:46,578] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:41:46,608] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:41:46,609] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:41:46,635] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:41:46,724] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:41:46,752] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 12871544832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:41:46,779] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:41:46,808] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:41:46,808] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:41:46,859] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:41:46,864] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:46,864] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:47,567] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:47,569] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:41:47,696] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:41:47,698] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:41:47,728] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:41:47,729] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:41:47,754] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:41:47,837] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:41:47,863] - [task:165] INFO -  {'uptime': '843', 'memoryTotal': 15466930176, 'memoryFree': 13013901312, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:41:47,889] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:41:47,917] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:41:47,918] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:41:47,968] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:41:47,971] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:47,971] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:48,619] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:48,620] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:41:48,742] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:41:48,744] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:41:48,773] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:41:48,774] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:41:48,798] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:41:48,881] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:41:48,907] - [task:165] INFO -  {'uptime': '844', 'memoryTotal': 15466930176, 'memoryFree': 13005725696, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:41:48,933] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:41:48,960] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:41:48,961] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:41:49,011] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:41:49,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:49,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:49,637] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:49,639] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:41:49,774] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:41:49,775] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:41:49,804] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:41:49,805] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:41:49,830] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:41:49,889] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:41:50,541] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:41:55,546] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:41:55,637] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:41:55,643] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:55,644] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:56,298] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:56,300] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:41:56,419] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:41:56,421] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:41:56,421] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:41:57,031] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:41:57,086] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:41:57,087] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:41:58,018] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:41:58,332] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:41:58,595] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:41:58,598] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 07:41:58,652] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:58,653] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:41:59,330] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:41:59,335] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:41:59,335] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:42:00,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:42:00,370] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:42:00,370] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:42:01,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:42:01,559] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:42:01,559] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:42:02,757] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:42:09,768] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:42:09,768] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.14916989833015, 'mem_free': 13009387520, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:42:09,768] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:42:09,769] - [basetestcase:467] INFO - Time to execute basesetup : 25.237191438674927
[2022-09-12 07:42:09,825] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:42:09,825] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:42:09,880] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:42:09,881] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:42:09,935] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:42:09,936] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:42:09,989] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:42:09,989] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:42:10,040] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:42:10,099] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:42:10,100] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:42:10,100] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:42:15,108] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:42:15,111] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:42:15,111] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:42:15,764] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:42:16,817] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:42:16,994] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:42:20,185] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:42:20,263] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:42:20,264] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:42:20,264] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:42:50,286] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:42:50,316] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:42:50,342] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:42:50,407] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.941675ms
[2022-09-12 07:42:50,407] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'e7a89283-50ff-4789-8937-91c9d0a63b31', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.941675ms', 'executionTime': '63.882866ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:42:50,408] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:42:50,433] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:42:50,458] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:42:51,202] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 742.314326ms
[2022-09-12 07:42:51,202] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:42:51,252] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:42:51,288] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:42:51,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.543157ms
[2022-09-12 07:42:51,510] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:42:51,549] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:42:51,568] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:42:51,569] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:42:51,590] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:42:51,655] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:42:52,467] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:42:52,494] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:42:52,547] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.512315ms
[2022-09-12 07:42:52,548] - [base_gsi:282] INFO - BUILD INDEX on default(`employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr`) USING GSI
[2022-09-12 07:42:53,579] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr`) USING GSI
[2022-09-12 07:42:53,605] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%60%29+USING+GSI
[2022-09-12 07:42:53,627] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.755436ms
[2022-09-12 07:42:53,666] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr'
[2022-09-12 07:42:53,697] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%27
[2022-09-12 07:42:53,707] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.580245ms
[2022-09-12 07:42:54,739] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr'
[2022-09-12 07:42:54,766] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%27
[2022-09-12 07:42:54,776] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.132493ms
[2022-09-12 07:42:55,807] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr'
[2022-09-12 07:42:55,833] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%27
[2022-09-12 07:42:55,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.646962ms
[2022-09-12 07:42:56,873] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:42:56,897] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 07:42:56,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.889027ms
[2022-09-12 07:42:56,902] - [base_gsi:504] INFO - {'requestID': '938b8021-c2cd-4018-a360-1b49390094cc', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr', 'index_id': 'd935c5a8203e1b99', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.889027ms', 'executionTime': '2.83176ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 07:42:56,902] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:42:56,903] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:56,903] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:42:56,903] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:42:56,904] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:56,904] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:56,904] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:56,904] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:56,965] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:57,008] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:42:57,034] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 07:42:57,205] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 157.282431ms
[2022-09-12 07:42:57,205] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:42:57,205] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:42:58,408] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-12 07:42:58,467] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-12 07:42:58,498] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:42:58,525] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 07:42:58,529] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.910173ms
[2022-09-12 07:42:58,529] - [base_gsi:504] INFO - {'requestID': '79358b62-5528-48fe-8c72-1f58849a859e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr', 'index_id': 'd935c5a8203e1b99', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.910173ms', 'executionTime': '1.847864ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 07:42:58,529] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:42:58,530] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:58,530] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:42:58,530] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:42:58,531] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:58,531] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:58,531] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:58,531] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:42:58,585] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:42:58,626] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:42:58,653] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 07:42:58,751] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 89.363234ms
[2022-09-12 07:42:58,751] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:42:58,752] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:42:59,944] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr ON default USING GSI
[2022-09-12 07:42:59,969] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr+ON+default+USING+GSI
[2022-09-12 07:43:00,033] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.220724ms
[2022-09-12 07:43:00,065] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr'
[2022-09-12 07:43:00,095] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec8c265fc6ff94d2fbf957752dc7f6bbbjoin_yr%27
[2022-09-12 07:43:00,104] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.116849ms
[2022-09-12 07:43:00,104] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1a5fdac1-2b4a-44af-9dba-4508880ca8da', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.116849ms', 'executionTime': '7.040962ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:43:00,218] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:43:00,221] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:00,221] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:00,955] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:01,011] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:43:01,011] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:43:01,124] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:43:01,181] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:43:01,184] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:01,184] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:01,896] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:01,956] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:43:01,956] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:43:02,081] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:43:02,138] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:43:02,139] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:43:02,139] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:43:02,170] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:43:02,198] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:43:02,208] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.710976ms
[2022-09-12 07:43:02,236] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:43:02,260] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:43:02,302] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.968788ms
[2022-09-12 07:43:02,369] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:43:02,369] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.086685811304, 'mem_free': 12823957504, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:43:02,370] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:43:02,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:02,375] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:03,134] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:03,139] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:03,139] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:03,867] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:03,871] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:03,872] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:05,077] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:05,091] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:05,091] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:06,360] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:14,356] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-12 07:43:14,508] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:43:15,645] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:43:15,674] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:43:15,674] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:43:15,736] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:43:15,792] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:43:15,846] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:43:15,847] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:43:15,929] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:43:15,930] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:43:15,956] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:16,089] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:43:16,089] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:16,117] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:43:16,144] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:43:16,144] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:16,173] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:43:16,203] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:43:16,203] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:16,232] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:43:16,260] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:43:16,261] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:16,288] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:43:16,289] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 07:43:16,289] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:43:16,289] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 91.811s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=forestdb -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'forestdb', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_11'}
[2022-09-12 07:43:16,373] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:16,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:17,079] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:17,117] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:43:17,199] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:43:17,199] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-12 07:43:17,200] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:43:17,401] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:43:17,429] - [task:165] INFO -  {'uptime': '938', 'memoryTotal': 15466930176, 'memoryFree': 12994736128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:43:17,455] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:43:17,455] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:43:17,455] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:43:17,487] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:43:17,521] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:43:17,521] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:43:17,551] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:43:17,552] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:43:17,552] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:43:17,552] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:43:17,600] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:43:17,604] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:17,604] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:18,307] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:18,308] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:43:18,436] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:43:18,437] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:43:18,466] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:43:18,467] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:43:18,493] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:43:18,579] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:43:18,608] - [task:165] INFO -  {'uptime': '938', 'memoryTotal': 15466930176, 'memoryFree': 12993056768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:43:18,635] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:43:18,664] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:43:18,665] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:43:18,718] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:43:18,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:18,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:19,454] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:19,455] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:43:19,592] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:43:19,593] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:43:19,627] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:43:19,628] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:43:19,656] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:43:19,749] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:43:19,779] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12993380352, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:43:19,807] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:43:19,836] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:43:19,837] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:43:19,894] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:43:19,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:19,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:20,636] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:20,637] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:43:20,779] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:43:20,780] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:43:20,811] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:43:20,811] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:43:20,839] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:43:20,927] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:43:20,956] - [task:165] INFO -  {'uptime': '934', 'memoryTotal': 15466930176, 'memoryFree': 12993183744, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:43:20,984] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:43:21,014] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:43:21,015] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:43:21,069] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:43:21,072] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:21,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:21,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:21,785] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:43:21,914] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:43:21,915] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:43:21,944] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:43:21,945] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:43:21,970] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:43:22,029] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:43:22,679] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:43:27,682] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:43:27,771] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:43:27,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:27,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:28,502] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:28,503] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:43:28,631] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:43:28,632] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:43:28,633] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:43:29,188] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:43:29,246] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 07:43:29,246] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:43:30,188] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:43:30,507] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:43:30,808] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:43:30,813] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 07:43:30,871] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:30,871] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:31,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:31,596] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:31,596] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:32,698] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:32,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:32,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:33,937] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:33,948] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:33,948] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:35,185] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:42,270] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:43:42,270] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.08821147478044, 'mem_free': 12990582784, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:43:42,270] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:43:42,271] - [basetestcase:467] INFO - Time to execute basesetup : 25.899391412734985
[2022-09-12 07:43:42,322] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:43:42,323] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:43:42,379] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:43:42,380] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:43:42,437] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:43:42,437] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:43:42,491] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:43:42,491] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:43:42,543] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:43:42,620] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:43:42,620] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:43:42,621] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:43:47,629] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:43:47,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:43:47,633] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:43:48,343] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:43:49,522] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:43:49,694] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:43:52,652] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:43:52,734] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:43:52,734] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:43:52,734] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:44:22,765] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:44:22,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:44:22,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:44:22,887] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.443514ms
[2022-09-12 07:44:22,887] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '585edee1-301a-4d15-9751-1d43dc878ee8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.443514ms', 'executionTime': '65.382484ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:44:22,888] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:44:22,913] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:44:22,943] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:44:23,662] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 717.335255ms
[2022-09-12 07:44:23,662] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:44:23,744] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:44:23,773] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:44:23,782] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.418491ms
[2022-09-12 07:44:23,999] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:24,038] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:44:24,054] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:44:24,064] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:44:24,084] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:44:24,148] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:24,953] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee01437b25750e477a88483d5237bd8b18join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 07:44:24,980] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee01437b25750e477a88483d5237bd8b18join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 07:44:25,053] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.798377ms
[2022-09-12 07:44:25,054] - [base_gsi:282] INFO - BUILD INDEX on default(`employee01437b25750e477a88483d5237bd8b18join_yr`) USING GSI
[2022-09-12 07:44:26,091] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee01437b25750e477a88483d5237bd8b18join_yr`) USING GSI
[2022-09-12 07:44:26,120] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee01437b25750e477a88483d5237bd8b18join_yr%60%29+USING+GSI
[2022-09-12 07:44:26,146] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.36366ms
[2022-09-12 07:44:26,190] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee01437b25750e477a88483d5237bd8b18join_yr'
[2022-09-12 07:44:26,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee01437b25750e477a88483d5237bd8b18join_yr%27
[2022-09-12 07:44:26,234] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.55983ms
[2022-09-12 07:44:27,265] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee01437b25750e477a88483d5237bd8b18join_yr'
[2022-09-12 07:44:27,293] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee01437b25750e477a88483d5237bd8b18join_yr%27
[2022-09-12 07:44:27,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.81927ms
[2022-09-12 07:44:28,335] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee01437b25750e477a88483d5237bd8b18join_yr'
[2022-09-12 07:44:28,360] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee01437b25750e477a88483d5237bd8b18join_yr%27
[2022-09-12 07:44:28,369] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.457339ms
[2022-09-12 07:44:29,400] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:44:29,427] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 07:44:29,431] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.828891ms
[2022-09-12 07:44:29,432] - [base_gsi:504] INFO - {'requestID': 'ad7deaa2-2999-4929-b207-2592da10701e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee01437b25750e477a88483d5237bd8b18join_yr', 'index_id': '74f348fe5af5bee1', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.828891ms', 'executionTime': '2.755399ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 07:44:29,432] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 07:44:29,432] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:44:29,433] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 07:44:29,433] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 07:44:29,433] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:44:29,433] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:44:29,433] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:44:29,434] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 07:44:29,494] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 07:44:29,535] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 07:44:29,562] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 07:44:29,750] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 174.831419ms
[2022-09-12 07:44:29,750] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 07:44:29,751] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 07:44:31,787] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 07:44:34,434] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:44:34,488] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 07:44:34,489] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:44:35,371] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:44:35,683] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:44:35,958] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:44:35,961] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 07:44:37,998] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:44:37,998] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:38,027] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:44:38,055] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:44:38,055] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:38,083] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:44:38,111] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:44:38,111] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:38,137] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:44:38,165] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:44:38,165] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:38,191] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:44:38,245] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:38,271] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee01437b25750e477a88483d5237bd8b18join_yr'
[2022-09-12 07:44:38,298] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee01437b25750e477a88483d5237bd8b18join_yr%27
[2022-09-12 07:44:38,344] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.446405ms
[2022-09-12 07:44:38,344] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7711199a-0f68-4967-9cee-b2fabec44066', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '44.446405ms', 'executionTime': '44.37853ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:44:38,371] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee01437b25750e477a88483d5237bd8b18join_yr'
[2022-09-12 07:44:38,396] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee01437b25750e477a88483d5237bd8b18join_yr%27
[2022-09-12 07:44:38,398] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.21084ms
[2022-09-12 07:44:38,399] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '781afc17-ac15-4988-aac2-182976671c7c', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.21084ms', 'executionTime': '1.153424ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:44:38,450] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:38,580] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:38,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:38,583] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:39,316] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:39,373] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:44:39,374] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:44:39,487] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:44:39,545] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:39,548] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:39,548] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:40,267] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:40,324] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:44:40,325] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:44:40,452] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:44:40,514] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:44:40,514] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:44:40,514] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:44:40,541] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:44:40,568] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:44:40,577] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.785065ms
[2022-09-12 07:44:40,577] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'fd26b7e8-c4c9-4f08-942b-52c96d9f642b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.785065ms', 'executionTime': '7.722991ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:44:40,577] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:44:40,603] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:44:40,629] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:44:40,632] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.283288ms
[2022-09-12 07:44:40,632] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '52d1e4ba-dab8-4815-a7e1-33c6ccae04b5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.283288ms', 'executionTime': '1.213904ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:44:40,688] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:44:40,688] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.36423308195561, 'mem_free': 12802396160, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:44:40,688] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:44:40,691] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:40,692] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:41,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:41,424] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:41,424] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:42,205] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:42,219] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:42,220] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:43,464] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:43,471] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:43,472] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:44,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:52,495] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 07:44:52,639] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:44:52,950] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:44:52,978] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:44:52,978] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:44:53,036] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:44:53,093] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:44:53,145] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:44:53,146] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:44:53,226] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:44:53,227] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:44:53,252] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:53,382] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:44:53,382] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:53,409] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:44:53,435] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:44:53,435] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:53,463] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:44:53,489] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:44:53,489] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:53,517] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:44:53,543] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:44:53,544] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:53,571] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:44:53,571] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 07:44:53,571] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:44:53,572] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 97.256s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_07-27-34/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-12 07:44:53,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:53,634] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:54,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:54,425] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:44:54,506] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 07:44:54,506] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-12 07:44:54,507] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 07:44:54,659] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 07:44:54,687] - [task:165] INFO -  {'uptime': '1036', 'memoryTotal': 15466930176, 'memoryFree': 12976648192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:44:54,713] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 07:44:54,713] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 07:44:54,714] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 07:44:54,742] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 07:44:54,779] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 07:44:54,779] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 07:44:54,807] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 07:44:54,808] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 07:44:54,808] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 07:44:54,808] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 07:44:54,856] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:44:54,859] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:54,859] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:55,595] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:55,596] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:44:55,735] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:44:55,736] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:44:55,771] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:44:55,772] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:44:55,800] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:44:55,884] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 07:44:55,911] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12975751168, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:44:55,937] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:44:55,964] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 07:44:55,965] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 07:44:56,016] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:44:56,018] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:56,019] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:56,709] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:56,711] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:44:56,846] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:44:56,847] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:44:56,877] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:44:56,878] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:44:56,904] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9001/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:44:56,992] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 07:44:57,020] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12968394752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:44:57,046] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:44:57,074] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 07:44:57,074] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 07:44:57,128] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:44:57,131] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:57,131] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:57,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:57,901] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:44:58,050] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:44:58,051] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:44:58,086] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:44:58,087] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:44:58,114] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9002/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:44:58,202] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 07:44:58,231] - [task:165] INFO -  {'uptime': '1034', 'memoryTotal': 15466930176, 'memoryFree': 12975247360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 07:44:58,258] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 07:44:58,288] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 07:44:58,288] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 07:44:58,344] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 07:44:58,347] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:44:58,348] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:44:59,109] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:44:59,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:44:59,253] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:44:59,254] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:44:59,287] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 07:44:59,288] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=forestdb
[2022-09-12 07:44:59,316] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9003/settings/indexes body: storageMode=forestdb headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"storageMode":"The value must be one of the following: [plasma,memory_optimized]"}}' auth: Administrator:asdasd
[2022-09-12 07:44:59,375] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 07:44:59,917] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:45:04,918] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 07:45:05,006] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 07:45:05,010] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:05,010] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:05,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:05,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 07:45:05,911] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 07:45:05,912] - [remote_util:5249] INFO - b'ok'
[2022-09-12 07:45:05,912] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 07:45:06,533] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 07:45:06,591] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 07:45:06,592] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 07:45:07,486] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:45:07,786] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:45:08,040] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 07:45:08,043] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-12 07:45:08,098] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:08,098] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:08,795] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:08,799] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:08,799] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:09,910] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:09,917] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:09,917] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:11,204] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:11,216] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:11,216] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:12,469] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:19,928] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:45:19,928] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.68702383671647, 'mem_free': 12973043712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:45:19,928] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:45:19,929] - [basetestcase:467] INFO - Time to execute basesetup : 26.297827005386353
[2022-09-12 07:45:19,981] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:45:19,981] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:45:20,036] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:45:20,037] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:45:20,091] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:45:20,092] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:45:20,149] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 07:45:20,149] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 07:45:20,204] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:45:20,266] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'forestdb'} set
[2022-09-12 07:45:20,271] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 07:45:20,272] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 07:45:25,281] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 07:45:25,284] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:45:25,284] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:45:26,018] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:45:27,229] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 07:45:27,412] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 07:45:30,321] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 07:45:30,399] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:45:30,399] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 07:45:30,399] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 07:46:00,420] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 07:46:00,449] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:46:00,476] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:46:00,545] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.41599ms
[2022-09-12 07:46:00,546] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'a7f5753e-4547-41f6-af7f-9448ca44088f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.41599ms', 'executionTime': '67.342981ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 07:46:00,546] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 07:46:00,572] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 07:46:00,597] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 07:46:01,292] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 693.54888ms
[2022-09-12 07:46:01,293] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 07:46:01,347] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:46:01,383] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:46:01,391] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.210337ms
[2022-09-12 07:46:01,591] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:46:01,625] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 07:46:01,646] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 07:46:01,646] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 07:46:01,659] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 07:46:01,735] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:46:01,864] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:46:01,867] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:01,868] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:02,696] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:02,758] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:46:02,759] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 07:46:02,887] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:46:02,944] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:46:02,947] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:02,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:03,719] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:03,776] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 07:46:03,777] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 07:46:03,904] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 07:46:03,961] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 07:46:03,961] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 07:46:03,961] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:46:03,986] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 07:46:04,011] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 07:46:04,021] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.326084ms
[2022-09-12 07:46:04,047] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 07:46:04,072] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 07:46:04,128] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.043972ms
[2022-09-12 07:46:04,199] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 07:46:04,199] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.943838997907708, 'mem_free': 12797734912, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 07:46:04,199] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 07:46:04,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:04,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:04,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:04,993] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:04,993] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:06,237] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:06,250] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:06,250] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:07,615] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:07,623] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 07:46:07,623] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 07:46:08,976] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 07:46:16,929] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-12 07:46:17,074] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 07:46:18,144] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 07:46:18,171] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 07:46:18,172] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 07:46:18,240] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:46:18,294] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:46:18,348] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 07:46:18,349] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 07:46:18,432] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 07:46:18,433] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 07:46:18,458] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:46:18,585] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 07:46:18,585] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:46:18,611] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 07:46:18,637] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 07:46:18,637] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:46:18,664] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 07:46:18,689] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 07:46:18,689] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:46:18,716] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 07:46:18,741] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 07:46:18,742] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 07:46:18,767] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 07:46:18,768] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-12 07:46:18,768] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 07:46:18,769] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.215s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91:  8452 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-12.09.2022-06.01.cfg
Build Log: make-12.09.2022-06.01.log
Server Log: logs-12.09.2022-06.01.tar.gz

Finished