Building

Started building at 2022/09/12 09:31:00
Using pegged server, 1969 build
Calculating base
Updating mirror
Basing run on 7.2.0-1969 97f96e53c
Updating tree for run 12.09.2022-09.31
query is at d5f057e, changes since last good build: none
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 571a538, changes since last good build: none
couchstore is at 803ec5f, changes since last good build: none
forestdb is at acba458, changes since last good build: none
kv_engine is at 9b8ee24, changes since last good build: none
Switching indexing to unstable
indexing is at 4da8ba0, changes since last good build: none
Switching plasma to unstable
plasma is at fb7c5c5, changes since last good build: 
fatal: Invalid revision range 1c5995aedb73edda33523300ee73b3da1e3ca5a2..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at dc1cc76, changes since last good build: none
Pulling in uncommitted change 178771 at refs/changes/71/178771/6
Total 6 (delta 1), reused 2 (delta 1)
[unstable d50535e6] MB-52790 verbose logging to detect auth error
 Author: Dhruvil Shah 
 Date: Fri Aug 12 11:01:54 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178774 at refs/changes/74/178774/16
Total 5 (delta 3), reused 5 (delta 3)
[unstable 86c0b359] MB-53361 updating go.mod
 Author: Dhruvil Shah 
 Date: Fri Aug 12 12:13:27 2022 +0530
 3 files changed, 644 insertions(+), 21 deletions(-)
 create mode 100644 go.sum
Pulling in uncommitted change 179711 at refs/changes/11/179711/2
Total 6 (delta 1), reused 1 (delta 1)
[unstable 6534dfbe] disabling TestPartitionDistributionWithReplica
 Author: Dhruvil Shah 
 Date: Thu Sep 8 13:13:02 2022 +0530
 1 file changed, 1 insertion(+)
Pulling in uncommitted change 179830 at refs/changes/30/179830/1
Total 4 (delta 2), reused 3 (delta 2)
[unstable 892e731] MB-52020: Fix RestoreShard on destination having no indexes
 Author: saptarshi.sen 
 Date: Sun Sep 11 16:59:58 2022 -0700
 2 files changed, 32 insertions(+), 1 deletion(-)
Pulling in uncommitted change 179831 at refs/changes/31/179831/1
Total 8 (delta 6), reused 8 (delta 6)
[unstable 9177ec5] MB-52020: Use uint64 type for ShardID
 Author: saptarshi.sen 
 Date: Sun Sep 11 17:12:12 2022 -0700
 2 files changed, 13 insertions(+), 13 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/12 10:21:11
Testing mode: sanity,unit,functional,integration
Using storage type: plasma
Setting ulimit to 200000

Simple Test

Sep 12 10:26:36 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 10:30:30 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 12 10:31:16 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 12 10:32:15 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 12 10:33:02 b" 'stop_on_failure': 'True'}"
Sep 12 10:33:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 10:33:02 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_1'}"
Sep 12 10:33:02 b'-->result: '
Sep 12 10:33:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 12 10:33:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 12 10:33:02 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_2'}"
Sep 12 10:33:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:33:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 12 10:33:02 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_3'}"
Sep 12 10:33:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:33:02 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:33:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 12 10:33:02 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_4'}"
Sep 12 10:33:02 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:33:02 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:33:02 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 10:43:02 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 12 10:43:49 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 12 10:47:36 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 10:51:33 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 10:54:15 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 12 10:54:15 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_5'}"
Sep 12 10:54:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 12 10:54:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 12 10:54:15 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_6'}"
Sep 12 10:54:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 10:54:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 12 10:54:15 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_7'}"
Sep 12 10:54:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 12 10:54:15 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 12 10:54:15 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-21-33/test_8'}"
Sep 12 10:54:15 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 12 10:54:15 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 12 10:54:15 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 12 10:54:15 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 12 10:54:16 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 10:54:16 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 12 10:54:16 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 12 10:54:16 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 12 10:54:16 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 12 10:54:16 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 12 10:54:16 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 12 10:54:16 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

/home/buildbot/bin/dotest: line 187: go: command not found
/home/buildbot/bin/dotest: line 190: go: command not found
/home/buildbot/bin/dotest: line 198: go: command not found
/home/buildbot/bin/dotest: line 201: go: command not found
/home/buildbot/bin/dotest: line 204: go: command not found
Starting server: attempt 1

Functional tests

/home/buildbot/bin/dotest: line 217: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
/home/buildbot/bin/dotest: line 222: go: command not found
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=plasma
Printing gsi_type=plasma
gsi_type=plasma
In here
-p makefile=True,gsi_type=plasma
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'plasma',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-12 10:57:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:08 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:08 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:08 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 10:57:11 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 10:57:17 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-12 10:57:29 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:29 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:30 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:31 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:36 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 10:57:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:57:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 10:57:36 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:57:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 10:57:37 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:57:37 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9003/pools/default error [Errno 111] Connection refused 
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 10:57:40 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:40 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-12 10:57:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '28', 'memoryTotal': 15466930176, 'memoryFree': 13542727680, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:57:41 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:41 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:57:42 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '25', 'memoryTotal': 15466930176, 'memoryFree': 1917378560, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:57:42 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:42 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:57:43 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '27', 'memoryTotal': 15466930176, 'memoryFree': 1791033344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:57:43 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:43 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:57:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '26', 'memoryTotal': 15466930176, 'memoryFree': 1691979776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:57:44 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:57:44 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:57:45 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 10:57:45 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 10:57:45 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-12 10:57:45 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:57:50 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 10:57:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 10:57:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-12 10:57:51 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 10:57:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:57:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:57:52 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 10:57:52 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-12 10:57:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:52 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:57:53 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 45.59982526296113, 'mem_free': 13408903168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 47.67686223983765
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 10:57:56 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 10:58:01 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 10:58:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:01 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:01 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:02 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 10:58:02 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:58:04 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 10:58:04 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:04 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 10:58:04 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 129.230459ms
2022-09-12 10:58:34 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'f6e8d712-9ffc-4cbd-b8e0-096c788d7782', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '129.230459ms', 'executionTime': '129.084188ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 10:58:34 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 745.493862ms
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.181071ms
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:35 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:58:36 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.012736ms
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 52.92561ms
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.617515352712366, 'mem_free': 13379436544, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:37 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:38 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:41 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 10:58:42 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 10:58:42 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 93.984s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1969-rel-EE-enterprise
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:42 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 17.57860334527503, 'mem_free': 13324587008, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:43 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:44 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:47 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 10:58:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:47 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 10:58:48 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-12 10:58:48 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '96', 'memoryTotal': 15466930176, 'memoryFree': 13324587008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-12 10:58:49 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:49 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '95', 'memoryTotal': 15466930176, 'memoryFree': 13355417600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:58:50 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '92', 'memoryTotal': 15466930176, 'memoryFree': 13355966464, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '91', 'memoryTotal': 15466930176, 'memoryFree': 13353754624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:51 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:58:52 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:58:52 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:58:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-12 10:58:52 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
2022-09-12 10:58:52 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-12 10:58:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 10:58:57 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-12 10:58:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-12 10:58:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:58:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:58:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:58:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-12 10:58:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-12 10:58:58 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-12 10:58:58 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-12 10:58:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-12 10:58:59 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.05 seconds to create bucket default
2022-09-12 10:58:59 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-12 10:59:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:59:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:59:00 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-12 10:59:00 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 10:59:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.14465406100516, 'mem_free': 13345288192, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 22.664424896240234
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'plasma'} set
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-12 10:59:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-12 10:59:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-12 10:59:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:10 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:11 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-12 10:59:11 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-12 10:59:13 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-12 10:59:13 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:13 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-12 10:59:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 65.358738ms
2022-09-12 10:59:43 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'fb4798a6-1aff-4a37-b5b8-1b9d92bd08e0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.358738ms', 'executionTime': '65.2385ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-12 10:59:43 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 706.052611ms
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 10.060596ms
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-12 10:59:44 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employee0fa4851f90ca49c7b7e2c3fde8aec776job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-12 10:59:45 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-12 10:59:45 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.118393ms
2022-09-12 10:59:45 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employee0fa4851f90ca49c7b7e2c3fde8aec776job_title) USING GSI
2022-09-12 10:59:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employee0fa4851f90ca49c7b7e2c3fde8aec776job_title) USING GSI
2022-09-12 10:59:46 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%29+USING+GSI
2022-09-12 10:59:46 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 20.653191ms
2022-09-12 10:59:47 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee0fa4851f90ca49c7b7e2c3fde8aec776job_title'
2022-09-12 10:59:48 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%27
2022-09-12 10:59:48 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 9.248673ms
2022-09-12 10:59:49 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee0fa4851f90ca49c7b7e2c3fde8aec776job_title'
2022-09-12 10:59:49 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%27
2022-09-12 10:59:49 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.89983ms
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 4.084785ms
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': '884a8293-6469-407e-a00a-43d2545c0712', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee0fa4851f90ca49c7b7e2c3fde8aec776job_title', 'index_id': 'c23b4d3c8deaf553', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '4.084785ms', 'executionTime': '4.02323ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:50 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-12 10:59:50 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 135.654492ms
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-12 10:59:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-12 10:59:52 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:52 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee0fa4851f90ca49c7b7e2c3fde8aec776job_title'
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%27
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.966422ms
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employee0fa4851f90ca49c7b7e2c3fde8aec776job_title ON default USING GSI
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employee0fa4851f90ca49c7b7e2c3fde8aec776job_title+ON+default+USING+GSI
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 34.638167ms
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employee0fa4851f90ca49c7b7e2c3fde8aec776job_title'
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee0fa4851f90ca49c7b7e2c3fde8aec776job_title%27
2022-09-12 10:59:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.90663ms
2022-09-12 10:59:53 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'e4611186-7591-47e2-9df2-9b89e32a1cca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.90663ms', 'executionTime': '5.845835ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:53 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.781087ms
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 51.856417ms
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.14842420583346, 'mem_free': 13233041408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:54 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-12 10:59:55 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-12 10:59:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-12 10:59:59 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 10:59:59 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-12 11:00:00 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-12 11:00:00 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-12 11:00:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-12 11:00:01 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-12 11:00:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 79.176s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_2'}
[2022-09-12 11:00:01,663] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:01,663] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:01,938] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:01,970] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:00:02,058] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:00:02,058] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-12 11:00:02,059] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:00:02,694] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:00:02,723] - [task:165] INFO -  {'uptime': '170', 'memoryTotal': 15466930176, 'memoryFree': 13260697600, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:00:02,752] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:00:02,753] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:00:02,753] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:00:02,789] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:00:02,823] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:00:02,824] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:00:02,854] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:00:02,855] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:00:02,855] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:00:02,855] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:00:02,907] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:00:02,910] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:02,911] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:03,175] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:03,176] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:00:03,244] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:00:03,245] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:00:03,275] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:03,302] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:03,334] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:00:03,467] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:00:03,494] - [task:165] INFO -  {'uptime': '165', 'memoryTotal': 15466930176, 'memoryFree': 13227642880, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:00:03,525] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:00:03,555] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:00:03,556] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:00:03,609] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:00:03,613] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:03,614] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:03,873] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:03,874] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:00:03,956] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:00:03,957] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:00:03,987] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:04,015] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:04,043] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:00:04,162] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:00:04,190] - [task:165] INFO -  {'uptime': '167', 'memoryTotal': 15466930176, 'memoryFree': 13227765760, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:00:04,216] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:00:04,246] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:00:04,246] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:00:04,302] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:00:04,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:04,305] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:04,593] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:04,595] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:00:04,667] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:00:04,669] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:00:04,700] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:04,728] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:04,760] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:00:04,885] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:00:04,913] - [task:165] INFO -  {'uptime': '166', 'memoryTotal': 15466930176, 'memoryFree': 13225439232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:00:04,942] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:00:04,973] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:00:04,973] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:00:05,028] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:00:05,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:05,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:05,276] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:05,277] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:00:05,345] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:00:05,345] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:00:05,375] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:05,401] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:00:05,431] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:00:05,523] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:00:06,182] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:00:11,187] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:00:11,274] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:00:11,279] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:11,279] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:11,554] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:11,555] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:00:11,624] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:00:11,625] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:00:11,625] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:00:12,654] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:00:12,709] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 11:00:12,710] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:00:13,606] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:00:13,929] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:00:14,203] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:00:14,261] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:00:14,321] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:14,321] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:14,576] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:14,580] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:14,580] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:14,853] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:14,858] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:14,858] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:15,123] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:15,127] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:15,127] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:15,580] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:19,022] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:00:19,022] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.39670114531732, 'mem_free': 13254692864, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:00:19,022] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:00:19,022] - [basetestcase:467] INFO - Time to execute basesetup : 17.361835956573486
[2022-09-12 11:00:19,074] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:00:19,075] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:00:19,131] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:00:19,131] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:00:19,188] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:00:19,188] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:00:19,243] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:00:19,244] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:00:19,299] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:00:19,361] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:00:19,361] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:00:19,361] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:00:24,371] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:00:24,374] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:00:24,374] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:00:24,647] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:00:25,588] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:00:25,757] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:00:28,413] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:00:28,436] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 11:00:29,101] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:00:29,101] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:00:29,102] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:00:59,131] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:00:59,160] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:00:59,186] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:00:59,252] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.849856ms
[2022-09-12 11:00:59,252] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b6566a95-ce3a-4b4c-9dd4-f05bf43b5f5b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.849856ms', 'executionTime': '64.723014ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:00:59,253] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:00:59,278] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:00:59,303] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:01:00,003] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 698.148357ms
[2022-09-12 11:01:00,003] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:01:00,054] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:01:00,093] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:01:00,101] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.107221ms
[2022-09-12 11:01:00,304] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:00,342] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:01:00,357] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:01:00,358] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:01:00,376] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:01:00,438] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:01,260] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef9f1aa4271e84f33927093eb5a063bdajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:01:01,286] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef9f1aa4271e84f33927093eb5a063bdajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:01:01,349] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.99669ms
[2022-09-12 11:01:01,350] - [base_gsi:282] INFO - BUILD INDEX on default(employeef9f1aa4271e84f33927093eb5a063bdajob_title) USING GSI
[2022-09-12 11:01:02,381] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeef9f1aa4271e84f33927093eb5a063bdajob_title) USING GSI
[2022-09-12 11:01:02,406] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeef9f1aa4271e84f33927093eb5a063bdajob_title%29+USING+GSI
[2022-09-12 11:01:02,436] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 27.852681ms
[2022-09-12 11:01:03,466] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef9f1aa4271e84f33927093eb5a063bdajob_title'
[2022-09-12 11:01:03,494] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef9f1aa4271e84f33927093eb5a063bdajob_title%27
[2022-09-12 11:01:03,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.345503ms
[2022-09-12 11:01:04,534] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef9f1aa4271e84f33927093eb5a063bdajob_title'
[2022-09-12 11:01:04,562] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef9f1aa4271e84f33927093eb5a063bdajob_title%27
[2022-09-12 11:01:04,569] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.835905ms
[2022-09-12 11:01:05,020] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 11:01:05,187] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:01:06,071] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:01:06,572] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:06,606] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:01:06,634] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 11:01:06,638] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.251971ms
[2022-09-12 11:01:06,639] - [task:3246] INFO - {'requestID': '5113152a-e8aa-481e-bc71-89f0a0437ad8', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef9f1aa4271e84f33927093eb5a063bdajob_title', 'index_id': '3fa34aed265f85a0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.251971ms', 'executionTime': '2.17961ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 11:01:06,639] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:06,639] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:06,640] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:01:06,640] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:01:06,640] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 11:01:06,640] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:01:06,641] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:01:06,641] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:01:06,641] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:01:07,641] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:07,671] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:01:07,698] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 11:01:07,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 119.565217ms
[2022-09-12 11:01:07,827] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:01:07,828] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:01:08,620] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:08,620] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:01:09,651] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef9f1aa4271e84f33927093eb5a063bdajob_title'
[2022-09-12 11:01:09,678] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef9f1aa4271e84f33927093eb5a063bdajob_title%27
[2022-09-12 11:01:09,687] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.617398ms
[2022-09-12 11:01:09,714] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeef9f1aa4271e84f33927093eb5a063bdajob_title ON default USING GSI
[2022-09-12 11:01:09,740] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeef9f1aa4271e84f33927093eb5a063bdajob_title+ON+default+USING+GSI
[2022-09-12 11:01:09,781] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.752944ms
[2022-09-12 11:01:09,820] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef9f1aa4271e84f33927093eb5a063bdajob_title'
[2022-09-12 11:01:09,846] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef9f1aa4271e84f33927093eb5a063bdajob_title%27
[2022-09-12 11:01:09,854] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.859641ms
[2022-09-12 11:01:09,854] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '46ac1230-1554-4e2f-b20d-47bca7588507', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.859641ms', 'executionTime': '5.739889ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:01:09,962] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:09,966] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:09,966] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:10,302] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:10,358] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:01:10,358] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:01:10,418] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:01:10,474] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:10,477] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:10,477] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:10,809] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:10,867] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:01:10,867] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:01:10,928] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:01:10,984] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:10,984] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:01:10,984] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:01:11,011] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:01:11,037] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:01:11,047] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.016867ms
[2022-09-12 11:01:11,075] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:01:11,101] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:01:11,161] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 57.383794ms
[2022-09-12 11:01:11,232] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:01:11,232] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.93034395515933, 'mem_free': 13143756800, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:01:11,233] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:01:11,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:11,237] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:11,597] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:11,602] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:11,602] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:11,990] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:11,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:11,996] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:12,537] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:12,547] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:12,548] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:13,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:17,404] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:01:17,656] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:01:18,184] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:01:18,211] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:01:18,212] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:01:18,267] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:01:18,320] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:01:18,375] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:01:18,377] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:01:18,458] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:01:18,459] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:01:18,484] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:18,619] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:01:18,619] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:18,650] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:01:18,675] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:01:18,675] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:18,702] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:01:18,728] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:01:18,728] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:18,755] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:01:18,787] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:01:18,788] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:18,815] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:01:18,816] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:01:18,816] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:01:18,816] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 77.213s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_3'}
[2022-09-12 11:01:18,903] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:18,903] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:19,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:19,255] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:01:19,338] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:01:19,338] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-12 11:01:19,339] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:01:19,934] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:01:19,964] - [task:165] INFO -  {'uptime': '247', 'memoryTotal': 15466930176, 'memoryFree': 13193863168, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:01:19,991] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:01:19,991] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:01:19,992] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:01:20,033] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:01:20,065] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:01:20,066] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:01:20,097] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:01:20,098] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:01:20,098] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:01:20,098] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:01:20,148] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:01:20,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:20,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:20,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:20,480] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:01:20,560] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:01:20,561] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:01:20,592] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:20,618] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:20,647] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:01:20,775] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:01:20,804] - [task:165] INFO -  {'uptime': '246', 'memoryTotal': 15466930176, 'memoryFree': 13195862016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:01:20,833] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:01:20,862] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:01:20,862] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:01:20,915] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:01:20,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:20,920] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:21,228] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:21,229] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:01:21,310] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:01:21,311] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:01:21,341] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:21,368] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:21,397] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:01:21,521] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:01:21,551] - [task:165] INFO -  {'uptime': '243', 'memoryTotal': 15466930176, 'memoryFree': 13195354112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:01:21,578] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:01:21,609] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:01:21,609] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:01:21,662] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:01:21,666] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:21,666] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:22,004] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:22,006] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:01:22,085] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:01:22,086] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:01:22,117] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:22,144] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:22,173] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:01:22,292] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:01:22,320] - [task:165] INFO -  {'uptime': '242', 'memoryTotal': 15466930176, 'memoryFree': 13193224192, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:01:22,347] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:01:22,377] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:01:22,378] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:01:22,432] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:01:22,434] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:22,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:22,781] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:22,782] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:01:22,863] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:01:22,864] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:01:22,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:22,922] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:01:22,953] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:01:23,055] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:01:23,593] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:01:28,598] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:01:28,692] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:01:28,697] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:28,697] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:29,045] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:29,046] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:01:29,127] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:01:29,128] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:01:29,128] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:01:30,194] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:01:30,252] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:01:30,253] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:01:31,143] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:01:31,451] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:01:31,717] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:01:31,720] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:01:31,776] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:31,776] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:32,097] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:32,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:32,101] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:32,401] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:32,405] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:32,406] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:32,719] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:32,769] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:32,769] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:33,308] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:37,428] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:01:37,428] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.14121977490123, 'mem_free': 13201629184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:01:37,429] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:01:37,429] - [basetestcase:467] INFO - Time to execute basesetup : 18.5278537273407
[2022-09-12 11:01:37,486] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:01:37,486] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:01:37,545] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:01:37,545] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:01:37,602] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:01:37,603] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:01:37,657] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:01:37,658] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:01:37,710] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:37,778] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:01:37,779] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:01:37,779] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:01:42,791] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:01:42,794] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:01:42,794] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:01:43,105] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:01:44,124] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:01:44,288] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:01:46,828] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:01:46,844] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 11:01:47,673] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:01:47,674] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:01:47,674] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:02:17,703] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:02:17,734] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:02:17,761] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:02:17,829] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.987347ms
[2022-09-12 11:02:17,830] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '92d5fc44-b792-450d-badb-200306f831b2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.987347ms', 'executionTime': '66.918099ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:02:17,830] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:02:17,856] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:02:17,886] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:02:18,613] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 725.330676ms
[2022-09-12 11:02:18,613] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:02:18,673] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:02:18,705] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:02:18,712] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.860361ms
[2022-09-12 11:02:18,917] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:18,959] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:02:18,973] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:02:18,973] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:02:18,990] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:02:19,063] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:19,869] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb829e1863a344c4bac9e5edfe392b99ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:02:19,896] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb829e1863a344c4bac9e5edfe392b99ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:02:19,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.189523ms
[2022-09-12 11:02:19,967] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb829e1863a344c4bac9e5edfe392b99ajob_title) USING GSI
[2022-09-12 11:02:20,998] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb829e1863a344c4bac9e5edfe392b99ajob_title) USING GSI
[2022-09-12 11:02:21,024] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb829e1863a344c4bac9e5edfe392b99ajob_title%29+USING+GSI
[2022-09-12 11:02:21,046] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.675902ms
[2022-09-12 11:02:22,079] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb829e1863a344c4bac9e5edfe392b99ajob_title'
[2022-09-12 11:02:22,105] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb829e1863a344c4bac9e5edfe392b99ajob_title%27
[2022-09-12 11:02:22,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.360379ms
[2022-09-12 11:02:23,147] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb829e1863a344c4bac9e5edfe392b99ajob_title'
[2022-09-12 11:02:23,175] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb829e1863a344c4bac9e5edfe392b99ajob_title%27
[2022-09-12 11:02:23,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.534041ms
[2022-09-12 11:02:23,643] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 11:02:23,817] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:02:24,484] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:02:25,187] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:25,220] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:02:25,245] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 11:02:25,250] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.887185ms
[2022-09-12 11:02:25,250] - [task:3246] INFO - {'requestID': '9ef5e6bd-62ff-4466-a472-80416284cdf2', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb829e1863a344c4bac9e5edfe392b99ajob_title', 'index_id': '5127d4c64ef19425', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.887185ms', 'executionTime': '2.824238ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 11:02:25,250] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:25,251] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:25,251] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:02:25,252] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:02:25,252] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 11:02:25,252] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:02:25,252] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:02:25,253] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:02:25,253] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:02:26,252] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:26,283] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:02:26,311] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 11:02:26,452] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 131.241679ms
[2022-09-12 11:02:26,452] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:02:26,453] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:02:27,227] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:27,227] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:02:28,259] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb829e1863a344c4bac9e5edfe392b99ajob_title'
[2022-09-12 11:02:28,284] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb829e1863a344c4bac9e5edfe392b99ajob_title%27
[2022-09-12 11:02:28,294] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.96214ms
[2022-09-12 11:02:28,319] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb829e1863a344c4bac9e5edfe392b99ajob_title ON default USING GSI
[2022-09-12 11:02:28,344] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb829e1863a344c4bac9e5edfe392b99ajob_title+ON+default+USING+GSI
[2022-09-12 11:02:28,390] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.802834ms
[2022-09-12 11:02:28,424] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb829e1863a344c4bac9e5edfe392b99ajob_title'
[2022-09-12 11:02:28,451] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb829e1863a344c4bac9e5edfe392b99ajob_title%27
[2022-09-12 11:02:28,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.703853ms
[2022-09-12 11:02:28,458] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b7e4be37-314d-4be1-9cea-5bff951f3d36', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.703853ms', 'executionTime': '5.63163ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:02:28,568] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:28,571] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:28,572] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:28,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:29,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:02:29,018] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:02:29,086] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:02:29,149] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:29,152] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:29,152] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:29,543] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:29,604] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:02:29,604] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:02:29,673] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:02:29,728] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:29,728] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:02:29,729] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:02:29,754] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:02:29,779] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:02:29,787] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.116415ms
[2022-09-12 11:02:29,812] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:02:29,837] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:02:29,888] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.025406ms
[2022-09-12 11:02:29,960] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:02:29,961] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.157763069786865, 'mem_free': 13099696128, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:02:29,961] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:02:29,965] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:29,965] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:30,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:30,385] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:30,385] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:30,935] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:30,946] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:30,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:31,651] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:31,658] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:31,658] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:32,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:36,727] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:02:36,867] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:02:37,771] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:02:37,799] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:02:37,799] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:02:37,853] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:02:37,907] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:02:37,964] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:02:37,965] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:02:38,056] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:02:38,057] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:02:38,083] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:38,215] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:02:38,215] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:38,243] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:02:38,268] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:02:38,269] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:38,295] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:02:38,320] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:02:38,320] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:38,350] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:02:38,376] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:02:38,376] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:38,404] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:02:38,404] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:02:38,404] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:02:38,405] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 79.556s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_4'}
[2022-09-12 11:02:38,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:38,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:38,892] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:38,924] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:02:39,008] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:02:39,008] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-12 11:02:39,009] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:02:39,535] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:02:39,564] - [task:165] INFO -  {'uptime': '327', 'memoryTotal': 15466930176, 'memoryFree': 13156835328, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:02:39,593] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:02:39,593] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:02:39,593] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:02:39,629] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:02:39,664] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:02:39,664] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:02:39,694] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:02:39,695] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:02:39,695] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:02:39,695] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:02:39,744] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:02:39,747] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:39,747] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:40,133] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:40,134] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:02:40,222] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:02:40,223] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:02:40,254] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:40,280] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:40,310] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:02:40,439] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:02:40,468] - [task:165] INFO -  {'uptime': '326', 'memoryTotal': 15466930176, 'memoryFree': 13155471360, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:02:40,496] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:02:40,527] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:02:40,527] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:02:40,580] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:02:40,583] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:40,583] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:40,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:40,986] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:02:41,069] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:02:41,070] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:02:41,100] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:41,126] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:41,157] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:02:41,282] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:02:41,310] - [task:165] INFO -  {'uptime': '323', 'memoryTotal': 15466930176, 'memoryFree': 13155721216, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:02:41,337] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:02:41,366] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:02:41,366] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:02:41,418] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:02:41,423] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:41,423] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:41,781] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:41,783] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:02:41,864] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:02:41,865] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:02:41,895] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:41,922] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:41,950] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:02:42,076] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:02:42,104] - [task:165] INFO -  {'uptime': '322', 'memoryTotal': 15466930176, 'memoryFree': 13154844672, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:02:42,132] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:02:42,161] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:02:42,161] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:02:42,215] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:02:42,218] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:42,218] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:42,653] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:42,654] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:02:42,739] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:02:42,741] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:02:42,774] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:42,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:02:42,834] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:02:42,921] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:02:43,480] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:02:48,486] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:02:48,588] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:02:48,593] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:48,593] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:48,966] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:48,967] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:02:49,056] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:02:49,058] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:02:49,058] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:02:50,069] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:02:50,127] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:02:50,128] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:02:51,060] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:02:51,374] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:02:51,642] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:02:51,644] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:02:51,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:51,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:52,074] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:52,079] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:52,079] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:52,467] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:52,472] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:52,473] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:53,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:53,020] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:02:53,020] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:02:53,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:02:58,358] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:02:58,358] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.09401936312148, 'mem_free': 13161844736, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:02:58,359] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:02:58,359] - [basetestcase:467] INFO - Time to execute basesetup : 19.855579137802124
[2022-09-12 11:02:58,415] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:02:58,415] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:02:58,471] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:02:58,472] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:02:58,526] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:02:58,526] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:02:58,579] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:02:58,579] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:02:58,630] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:02:58,692] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:02:58,693] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:02:58,694] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:03:03,706] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:03:03,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:03,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:04,085] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:05,111] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:03:05,281] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:03:07,744] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:03:07,759] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-12 11:03:08,445] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:08,445] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:03:08,446] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:03:38,470] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:03:38,500] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:03:38,525] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:03:38,592] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.674129ms
[2022-09-12 11:03:38,592] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f07f80d2-3961-40e0-bd7c-dcf0b3a5e55f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.674129ms', 'executionTime': '64.60946ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:03:38,592] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:03:38,618] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:03:38,643] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:03:39,326] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.328043ms
[2022-09-12 11:03:39,327] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:03:39,380] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:03:39,419] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:03:39,427] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.617644ms
[2022-09-12 11:03:39,652] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:39,690] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:03:39,708] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:03:39,709] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:03:39,728] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:03:39,804] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:40,612] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee120dc85760e742a194fca7461b384a9djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:03:40,638] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee120dc85760e742a194fca7461b384a9djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:03:40,686] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.063083ms
[2022-09-12 11:03:40,687] - [base_gsi:282] INFO - BUILD INDEX on default(employee120dc85760e742a194fca7461b384a9djob_title) USING GSI
[2022-09-12 11:03:41,717] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee120dc85760e742a194fca7461b384a9djob_title) USING GSI
[2022-09-12 11:03:41,742] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee120dc85760e742a194fca7461b384a9djob_title%29+USING+GSI
[2022-09-12 11:03:41,768] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.167676ms
[2022-09-12 11:03:42,800] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee120dc85760e742a194fca7461b384a9djob_title'
[2022-09-12 11:03:42,827] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee120dc85760e742a194fca7461b384a9djob_title%27
[2022-09-12 11:03:42,837] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.229024ms
[2022-09-12 11:03:43,867] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee120dc85760e742a194fca7461b384a9djob_title'
[2022-09-12 11:03:43,893] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee120dc85760e742a194fca7461b384a9djob_title%27
[2022-09-12 11:03:43,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.583912ms
[2022-09-12 11:03:44,389] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 11:03:44,558] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:03:45,426] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:03:45,611] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-12 11:03:45,611] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:03:45,779] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-12 11:03:45,779] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-12 11:03:45,780] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (3799843868, 0, b'')
[2022-09-12 11:03:45,904] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:45,931] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:03:45,961] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 11:03:45,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.693871ms
[2022-09-12 11:03:45,967] - [task:3246] INFO - {'requestID': 'f59401b6-603d-4d6c-8dea-aa2121f4651d', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee120dc85760e742a194fca7461b384a9djob_title', 'index_id': '169a9802b595d4fa', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.693871ms', 'executionTime': '3.619823ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 11:03:45,968] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:45,968] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:45,968] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:03:45,969] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:03:45,969] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 11:03:45,970] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:03:45,970] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:03:45,970] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:03:45,970] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:03:46,969] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:47,001] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:03:47,027] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 11:03:47,152] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 116.741721ms
[2022-09-12 11:03:47,153] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:03:47,153] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:03:47,921] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:47,921] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:03:48,954] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee120dc85760e742a194fca7461b384a9djob_title'
[2022-09-12 11:03:48,979] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee120dc85760e742a194fca7461b384a9djob_title%27
[2022-09-12 11:03:48,989] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.524295ms
[2022-09-12 11:03:49,015] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee120dc85760e742a194fca7461b384a9djob_title ON default USING GSI
[2022-09-12 11:03:49,041] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee120dc85760e742a194fca7461b384a9djob_title+ON+default+USING+GSI
[2022-09-12 11:03:49,095] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 51.871388ms
[2022-09-12 11:03:49,129] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee120dc85760e742a194fca7461b384a9djob_title'
[2022-09-12 11:03:49,156] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee120dc85760e742a194fca7461b384a9djob_title%27
[2022-09-12 11:03:49,162] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.92871ms
[2022-09-12 11:03:49,163] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f0ff00cf-2ab9-4c7f-89c3-39abb82808a1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.92871ms', 'executionTime': '4.864238ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:03:49,275] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:49,280] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:49,282] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:49,748] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:49,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:03:49,805] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:03:49,884] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:03:49,942] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:49,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:49,945] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:50,430] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:50,492] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:03:50,493] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:03:50,571] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:03:50,631] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:03:50,632] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:03:50,632] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:03:50,660] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:03:50,687] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:03:50,696] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.178775ms
[2022-09-12 11:03:50,726] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:03:50,754] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:03:50,789] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.007629ms
[2022-09-12 11:03:50,872] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:03:50,872] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.356552541322108, 'mem_free': 13049778176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:03:50,873] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:03:50,877] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:50,878] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:51,354] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:51,359] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:51,359] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:52,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:52,045] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:52,045] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:52,800] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:52,811] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:03:52,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:03:53,612] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:03:58,378] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:03:58,525] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:03:59,606] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:03:59,638] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:03:59,638] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:03:59,704] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:03:59,763] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:03:59,818] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:03:59,820] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:03:59,904] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:03:59,905] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:03:59,933] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,070] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:04:00,070] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,097] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:04:00,123] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:04:00,123] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,151] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:04:00,178] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:04:00,178] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,205] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:04:00,233] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:04:00,233] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,262] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:04:00,262] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:04:00,262] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:04:00,263] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 81.819s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_5'}
[2022-09-12 11:04:00,371] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:00,371] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:00,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:00,859] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:04:00,942] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:04:00,943] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-12 11:04:00,943] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:04:01,399] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:04:01,429] - [task:165] INFO -  {'uptime': '409', 'memoryTotal': 15466930176, 'memoryFree': 13121175552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:04:01,456] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:04:01,457] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:04:01,457] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:04:01,490] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:04:01,529] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:04:01,530] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:04:01,559] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:04:01,560] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:04:01,560] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:04:01,560] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:04:01,611] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:04:01,614] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:01,615] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:02,071] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:02,072] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:04:02,169] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:04:02,171] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:04:02,203] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:02,231] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:02,263] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:04:02,392] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:04:02,422] - [task:165] INFO -  {'uptime': '406', 'memoryTotal': 15466930176, 'memoryFree': 13120843776, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:04:02,449] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:04:02,479] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:04:02,479] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:04:02,535] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:04:02,538] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:02,538] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:02,995] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:02,996] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:04:03,095] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:04:03,096] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:04:03,128] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:03,157] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:03,189] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:04:03,314] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:04:03,345] - [task:165] INFO -  {'uptime': '403', 'memoryTotal': 15466930176, 'memoryFree': 13120819200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:04:03,373] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:04:03,402] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:04:03,402] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:04:03,453] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:04:03,458] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:03,458] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:03,868] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:03,869] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:04:03,964] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:04:03,965] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:04:03,995] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:04,021] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:04,050] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:04:04,167] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:04:04,196] - [task:165] INFO -  {'uptime': '402', 'memoryTotal': 15466930176, 'memoryFree': 13122048000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:04:04,224] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:04:04,252] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:04:04,252] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:04:04,305] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:04:04,309] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:04,309] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:04,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:04,772] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:04:04,871] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:04:04,872] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:04:04,904] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:04,933] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:04:04,963] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:04:05,056] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:04:05,600] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:04:10,606] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:04:10,694] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:04:10,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:10,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:11,147] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:11,148] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:04:11,239] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:04:11,240] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:04:11,241] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:04:12,181] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:04:12,235] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 11:04:12,236] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:04:13,167] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:04:13,503] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:04:13,780] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:04:13,783] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:04:13,839] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:13,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:14,282] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:14,287] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:14,287] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:14,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:14,738] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:14,738] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:15,497] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:15,505] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:15,505] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:16,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:21,381] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:04:21,382] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 22.85429600820119, 'mem_free': 13122355200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:04:21,382] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:04:21,382] - [basetestcase:467] INFO - Time to execute basesetup : 21.01375699043274
[2022-09-12 11:04:21,435] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:04:21,435] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:04:21,491] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:04:21,491] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:04:21,547] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:04:21,547] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:04:21,603] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:04:21,603] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:04:21,656] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:04:21,716] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:04:21,716] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:04:21,717] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:04:26,724] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:04:26,727] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:04:26,727] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:04:27,164] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:04:28,235] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:04:28,416] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:04:30,794] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:04:30,812] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 11:04:31,854] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:04:31,854] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:04:31,854] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:05:01,883] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:05:01,914] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:05:01,942] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:05:02,014] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.173387ms
[2022-09-12 11:05:02,014] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '79d368d7-46d0-4601-840a-e11dac43da92', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.173387ms', 'executionTime': '70.112773ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:05:02,015] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:05:02,043] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:05:02,070] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:05:02,778] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 705.506766ms
[2022-09-12 11:05:02,778] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:05:02,832] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:05:02,866] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:05:02,875] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.821188ms
[2022-09-12 11:05:03,074] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:03,119] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:05:03,144] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:05:03,145] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:05:03,167] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:05:03,246] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:04,034] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeee9f24595e05a4d4799cac496a363c6bdjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:05:04,061] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeee9f24595e05a4d4799cac496a363c6bdjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:05:04,134] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.274438ms
[2022-09-12 11:05:04,134] - [base_gsi:282] INFO - BUILD INDEX on default(employeee9f24595e05a4d4799cac496a363c6bdjob_title) USING GSI
[2022-09-12 11:05:05,165] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeee9f24595e05a4d4799cac496a363c6bdjob_title) USING GSI
[2022-09-12 11:05:05,193] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeee9f24595e05a4d4799cac496a363c6bdjob_title%29+USING+GSI
[2022-09-12 11:05:05,215] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.307423ms
[2022-09-12 11:05:06,247] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee9f24595e05a4d4799cac496a363c6bdjob_title'
[2022-09-12 11:05:06,273] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee9f24595e05a4d4799cac496a363c6bdjob_title%27
[2022-09-12 11:05:06,283] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.924237ms
[2022-09-12 11:05:07,313] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee9f24595e05a4d4799cac496a363c6bdjob_title'
[2022-09-12 11:05:07,339] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee9f24595e05a4d4799cac496a363c6bdjob_title%27
[2022-09-12 11:05:07,348] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.020997ms
[2022-09-12 11:05:07,825] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:05:07,998] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:05:11,640] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:05:12,355] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:12,387] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:05:12,413] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 11:05:12,418] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.603496ms
[2022-09-12 11:05:12,418] - [task:3246] INFO - {'requestID': 'bbc9794e-de80-40bf-bc6d-9de9fa793b82', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeee9f24595e05a4d4799cac496a363c6bdjob_title', 'index_id': '158b71365fe2efb2', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.603496ms', 'executionTime': '3.540744ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 11:05:12,419] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:12,419] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:12,419] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:05:12,420] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:05:12,420] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 11:05:12,420] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:05:12,421] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:05:12,421] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:05:12,421] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:05:13,420] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:13,450] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:05:13,478] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 11:05:13,677] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 184.099814ms
[2022-09-12 11:05:13,677] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:05:13,678] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:05:15,274] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:15,275] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:05:16,306] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee9f24595e05a4d4799cac496a363c6bdjob_title'
[2022-09-12 11:05:16,331] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee9f24595e05a4d4799cac496a363c6bdjob_title%27
[2022-09-12 11:05:16,341] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.537656ms
[2022-09-12 11:05:16,366] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeee9f24595e05a4d4799cac496a363c6bdjob_title ON default USING GSI
[2022-09-12 11:05:16,391] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeee9f24595e05a4d4799cac496a363c6bdjob_title+ON+default+USING+GSI
[2022-09-12 11:05:16,446] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 52.743959ms
[2022-09-12 11:05:16,485] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeee9f24595e05a4d4799cac496a363c6bdjob_title'
[2022-09-12 11:05:16,516] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeee9f24595e05a4d4799cac496a363c6bdjob_title%27
[2022-09-12 11:05:16,523] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.047555ms
[2022-09-12 11:05:16,524] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'bfb816b9-a403-4252-99f6-84385cba130d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.047555ms', 'executionTime': '5.917394ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:05:16,631] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:16,635] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:16,635] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:17,184] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:17,244] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:05:17,244] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:05:17,338] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:05:17,396] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:17,399] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:17,400] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:17,960] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:18,020] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:05:18,020] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:05:18,112] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:05:18,169] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:18,170] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:05:18,170] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:05:18,197] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:05:18,227] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:05:18,239] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.531892ms
[2022-09-12 11:05:18,267] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:05:18,292] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:05:18,334] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.665531ms
[2022-09-12 11:05:18,431] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:05:18,431] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.10088255448732, 'mem_free': 12937789440, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:05:18,431] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:05:18,436] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:18,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:18,985] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:18,991] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:18,991] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:19,541] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:19,546] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:19,546] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:20,443] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:20,454] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:20,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:21,382] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:27,456] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:05:27,600] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:05:28,824] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:05:28,855] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:05:28,856] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:05:28,916] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:05:28,970] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:05:29,026] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:05:29,027] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:05:29,110] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:05:29,112] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:05:29,140] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:29,276] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:05:29,276] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:29,303] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:05:29,328] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:05:29,329] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:29,356] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:05:29,381] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:05:29,381] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:29,408] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:05:29,433] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:05:29,433] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:29,459] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:05:29,460] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:05:29,460] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:05:29,460] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 89.143s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_6'}
[2022-09-12 11:05:29,566] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:29,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:30,091] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:30,124] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:05:30,202] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:05:30,202] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-12 11:05:30,203] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:05:30,593] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:05:30,620] - [task:165] INFO -  {'uptime': '498', 'memoryTotal': 15466930176, 'memoryFree': 13048262656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:05:30,646] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:05:30,646] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:05:30,646] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:05:30,675] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:05:30,711] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:05:30,712] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:05:30,741] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:05:30,742] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:05:30,742] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:05:30,742] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:05:30,789] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:05:30,793] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:30,793] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:31,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:31,291] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:05:31,393] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:05:31,395] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:05:31,424] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:31,450] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:31,480] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:05:31,603] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:05:31,629] - [task:165] INFO -  {'uptime': '497', 'memoryTotal': 15466930176, 'memoryFree': 13049192448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:05:31,655] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:05:31,685] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:05:31,686] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:05:31,736] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:05:31,739] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:31,739] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:32,250] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:32,251] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:05:32,361] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:05:32,362] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:05:32,394] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:32,422] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:32,452] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:05:32,568] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:05:32,595] - [task:165] INFO -  {'uptime': '494', 'memoryTotal': 15466930176, 'memoryFree': 13049131008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:05:32,626] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:05:32,655] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:05:32,655] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:05:32,711] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:05:32,717] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:32,718] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:33,236] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:33,237] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:05:33,344] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:05:33,346] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:05:33,377] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:33,403] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:33,432] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:05:33,560] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:05:33,587] - [task:165] INFO -  {'uptime': '493', 'memoryTotal': 15466930176, 'memoryFree': 13049053184, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:05:33,613] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:05:33,643] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:05:33,643] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:05:33,694] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:05:33,697] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:33,697] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:34,257] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:34,259] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:05:34,364] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:05:34,364] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:05:34,395] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:34,423] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:05:34,454] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:05:34,546] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:05:35,096] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:05:40,102] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:05:40,187] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:05:40,191] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:40,192] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:40,740] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:40,742] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:05:40,852] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:05:40,853] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:05:40,854] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:05:41,682] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:05:41,738] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:05:41,738] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:05:42,761] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:05:43,084] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:05:43,386] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:05:43,389] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:05:43,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:43,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:44,003] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:44,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:44,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:44,712] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:44,721] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:44,721] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:45,609] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:45,620] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:45,621] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:46,510] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:52,202] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:05:52,202] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 24.96808898887639, 'mem_free': 13049163776, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:05:52,202] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:05:52,203] - [basetestcase:467] INFO - Time to execute basesetup : 22.639113664627075
[2022-09-12 11:05:52,257] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:05:52,257] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:05:52,312] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:05:52,312] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:05:52,367] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:05:52,368] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:05:52,423] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:05:52,423] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:05:52,475] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:05:52,536] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:05:52,537] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:05:52,537] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:05:57,547] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:05:57,549] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:05:57,550] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:05:58,059] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:05:59,184] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:05:59,365] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:06:01,592] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:06:01,611] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-12 11:06:03,109] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:03,109] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:06:03,109] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:06:33,139] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:06:33,170] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:06:33,197] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:06:33,269] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.246156ms
[2022-09-12 11:06:33,269] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1e99bb4b-5656-4df0-af53-56f99299eee7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.246156ms', 'executionTime': '69.183447ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:06:33,269] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:06:33,297] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:06:33,323] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:06:34,013] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 687.371219ms
[2022-09-12 11:06:34,013] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:06:34,074] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:06:34,110] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:06:34,118] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.944528ms
[2022-09-12 11:06:34,312] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:34,350] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:06:34,363] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:06:34,364] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:06:34,380] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:06:34,459] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:35,279] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeb19040ef55234bdd9dff67cbeaa71892job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:06:35,306] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeb19040ef55234bdd9dff67cbeaa71892job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:06:35,364] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.725966ms
[2022-09-12 11:06:35,364] - [base_gsi:282] INFO - BUILD INDEX on default(employeeb19040ef55234bdd9dff67cbeaa71892job_title) USING GSI
[2022-09-12 11:06:36,394] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeb19040ef55234bdd9dff67cbeaa71892job_title) USING GSI
[2022-09-12 11:06:36,420] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeb19040ef55234bdd9dff67cbeaa71892job_title%29+USING+GSI
[2022-09-12 11:06:36,447] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 23.381412ms
[2022-09-12 11:06:37,481] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb19040ef55234bdd9dff67cbeaa71892job_title'
[2022-09-12 11:06:37,507] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb19040ef55234bdd9dff67cbeaa71892job_title%27
[2022-09-12 11:06:37,518] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.220279ms
[2022-09-12 11:06:38,549] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb19040ef55234bdd9dff67cbeaa71892job_title'
[2022-09-12 11:06:38,577] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb19040ef55234bdd9dff67cbeaa71892job_title%27
[2022-09-12 11:06:38,586] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.663983ms
[2022-09-12 11:06:38,972] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-12 11:06:39,147] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:06:40,280] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:06:40,901] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-12 11:06:41,073] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:06:42,381] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:06:43,017] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:06:43,184] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:06:46,669] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:06:47,627] - [task:3236] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:47,659] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:06:47,685] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-12 11:06:47,689] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.828906ms
[2022-09-12 11:06:47,689] - [task:3246] INFO - {'requestID': 'b1d495f7-649d-4412-9cd0-87eabfeb2c92', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeb19040ef55234bdd9dff67cbeaa71892job_title', 'index_id': '66860871b5e1ec8c', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.828906ms', 'executionTime': '2.770319ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-12 11:06:47,689] - [task:3247] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:47,690] - [task:3277] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:47,690] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:06:47,691] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:06:47,691] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-12 11:06:47,691] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:06:47,691] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:06:47,692] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:06:47,692] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-12 11:06:48,691] - [task:3236] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:48,722] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-12 11:06:48,748] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-12 11:06:48,962] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 199.920622ms
[2022-09-12 11:06:48,962] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:06:48,963] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:06:50,509] - [task:3247] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:50,509] - [task:3277] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-12 11:06:51,540] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb19040ef55234bdd9dff67cbeaa71892job_title'
[2022-09-12 11:06:51,566] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb19040ef55234bdd9dff67cbeaa71892job_title%27
[2022-09-12 11:06:51,576] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.159708ms
[2022-09-12 11:06:51,602] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeb19040ef55234bdd9dff67cbeaa71892job_title ON default USING GSI
[2022-09-12 11:06:51,627] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeb19040ef55234bdd9dff67cbeaa71892job_title+ON+default+USING+GSI
[2022-09-12 11:06:51,685] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.321153ms
[2022-09-12 11:06:51,721] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeb19040ef55234bdd9dff67cbeaa71892job_title'
[2022-09-12 11:06:51,749] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeb19040ef55234bdd9dff67cbeaa71892job_title%27
[2022-09-12 11:06:51,761] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.994933ms
[2022-09-12 11:06:51,762] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b7b07255-a10d-4754-b85d-8c1b4c8a89f9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '9.994933ms', 'executionTime': '9.919296ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:06:51,870] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:51,874] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:51,874] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:52,556] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:06:52,613] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:06:52,613] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:06:52,717] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:06:52,779] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:52,782] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:52,783] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:53,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:06:53,497] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:06:53,498] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:06:53,607] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:06:53,664] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:06:53,664] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:06:53,665] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:06:53,691] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:06:53,717] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:06:53,729] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.308679ms
[2022-09-12 11:06:53,761] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:06:53,789] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:06:53,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 35.903267ms
[2022-09-12 11:06:53,902] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:06:53,902] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.14878144805425, 'mem_free': 12858683392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:06:53,902] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:06:53,909] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:53,909] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:54,592] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:06:54,597] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:54,597] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:55,234] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:06:55,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:55,239] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:56,136] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:06:56,148] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:06:56,148] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:06:57,244] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:05,040] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:07:05,186] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:07:06,316] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:07:06,352] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:07:06,352] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:07:06,413] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:07:06,466] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:07:06,520] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:07:06,521] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:07:06,602] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:07:06,603] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:07:06,630] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:06,761] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:07:06,762] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:06,790] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:07:06,817] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:07:06,817] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:06,845] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:07:06,870] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:07:06,871] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:06,899] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:07:06,928] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:07:06,928] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:06,957] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:07:06,957] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-12 11:07:06,958] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:07:06,958] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 97.452s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_7'}
[2022-09-12 11:07:07,101] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:07,102] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:07,747] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:07,781] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:07:07,863] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:07:07,863] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-12 11:07:07,864] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:07:08,129] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:07:08,156] - [task:165] INFO -  {'uptime': '595', 'memoryTotal': 15466930176, 'memoryFree': 12978376704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:07:08,183] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:07:08,183] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:07:08,184] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:07:08,220] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:07:08,252] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:07:08,253] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:07:08,280] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:07:08,281] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:07:08,281] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:07:08,281] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:07:08,329] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:07:08,333] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:08,333] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:08,965] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:08,967] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:07:09,091] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:07:09,092] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:07:09,123] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:09,153] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:09,186] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:07:09,321] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:07:09,350] - [task:165] INFO -  {'uptime': '592', 'memoryTotal': 15466930176, 'memoryFree': 12977057792, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:07:09,377] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:07:09,407] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:07:09,407] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:07:09,460] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:07:09,463] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:09,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:10,092] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:10,093] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:07:10,212] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:07:10,214] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:07:10,245] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:10,270] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:10,300] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:07:10,425] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:07:10,452] - [task:165] INFO -  {'uptime': '594', 'memoryTotal': 15466930176, 'memoryFree': 12976513024, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:07:10,478] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:07:10,507] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:07:10,507] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:07:10,560] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:07:10,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:10,565] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:11,184] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:11,186] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:07:11,306] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:07:11,307] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:07:11,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:11,368] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:11,396] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:07:11,514] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:07:11,541] - [task:165] INFO -  {'uptime': '593', 'memoryTotal': 15466930176, 'memoryFree': 12977815552, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:07:11,567] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:07:11,596] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:07:11,596] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:07:11,648] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:07:11,651] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:11,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:12,223] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:12,224] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:07:12,345] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:07:12,346] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:07:12,376] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:12,402] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:07:12,430] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:07:12,517] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:07:13,176] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:07:18,178] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:07:18,265] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:07:18,271] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:18,271] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:18,929] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:18,930] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:07:19,053] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:07:19,054] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:07:19,055] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:07:19,651] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:07:19,706] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 11:07:19,706] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:07:20,636] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:07:20,950] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:07:21,209] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:07:21,211] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 11:07:21,266] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:21,267] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:21,901] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:21,906] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:21,906] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:22,779] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:22,787] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:22,787] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:23,880] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:23,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:23,890] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:24,997] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:31,491] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:07:31,492] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.12553884854382, 'mem_free': 12988678144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:07:31,492] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:07:31,492] - [basetestcase:467] INFO - Time to execute basesetup : 24.392894506454468
[2022-09-12 11:07:31,546] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:07:31,546] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:07:31,601] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:07:31,602] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:07:31,660] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:07:31,660] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:07:31,717] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:07:31,718] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:07:31,769] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:07:31,836] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:07:31,836] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:07:31,837] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:07:36,846] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:07:36,850] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:07:36,850] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:07:37,487] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:07:38,531] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:07:38,803] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:07:40,948] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:07:41,030] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:07:41,030] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:07:41,031] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:08:11,036] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:08:11,065] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:08:11,091] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:08:11,161] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.710974ms
[2022-09-12 11:08:11,161] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7be578ed-983e-4a13-a7a8-1822d545c2ac', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.710974ms', 'executionTime': '67.652054ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:08:11,162] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:08:11,188] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:08:11,214] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:08:11,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 674.991959ms
[2022-09-12 11:08:11,892] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:08:11,963] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:08:11,998] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:08:12,006] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.613255ms
[2022-09-12 11:08:12,219] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:08:12,257] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:08:12,275] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:08:12,276] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:08:12,284] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:08:12,285] - [base_gsi:326] INFO - []
[2022-09-12 11:08:13,183] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee37e50ea445874dfdbbd3a3dd1c7074a3job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:08:13,210] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:08:13,287] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 75.14553ms
[2022-09-12 11:08:13,323] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:08:13,349] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:08:13,399] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 47.57796ms
[2022-09-12 11:08:13,400] - [base_gsi:282] INFO - BUILD INDEX on default(employee37e50ea445874dfdbbd3a3dd1c7074a3job_title,employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr) USING GSI
[2022-09-12 11:08:14,434] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee37e50ea445874dfdbbd3a3dd1c7074a3job_title,employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr) USING GSI
[2022-09-12 11:08:14,469] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%2Cemployee37e50ea445874dfdbbd3a3dd1c7074a3join_yr%29+USING+GSI
[2022-09-12 11:08:14,513] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.277518ms
[2022-09-12 11:08:15,543] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3job_title'
[2022-09-12 11:08:15,571] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%27
[2022-09-12 11:08:15,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 9.865291ms
[2022-09-12 11:08:16,615] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3job_title'
[2022-09-12 11:08:16,640] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%27
[2022-09-12 11:08:16,650] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.665315ms
[2022-09-12 11:08:16,675] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr'
[2022-09-12 11:08:16,700] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr%27
[2022-09-12 11:08:16,703] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.446373ms
[2022-09-12 11:08:17,733] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3job_title'
[2022-09-12 11:08:17,758] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%27
[2022-09-12 11:08:17,768] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.802237ms
[2022-09-12 11:08:17,793] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee37e50ea445874dfdbbd3a3dd1c7074a3job_title ON default USING GSI
[2022-09-12 11:08:17,818] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee37e50ea445874dfdbbd3a3dd1c7074a3job_title+ON+default+USING+GSI
[2022-09-12 11:08:17,861] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.902513ms
[2022-09-12 11:08:17,892] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr'
[2022-09-12 11:08:17,923] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr%27
[2022-09-12 11:08:17,932] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.120259ms
[2022-09-12 11:08:17,961] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr ON default USING GSI
[2022-09-12 11:08:17,986] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr+ON+default+USING+GSI
[2022-09-12 11:08:18,032] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.884699ms
[2022-09-12 11:08:18,066] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3job_title'
[2022-09-12 11:08:18,092] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3job_title%27
[2022-09-12 11:08:18,100] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.342372ms
[2022-09-12 11:08:18,101] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'f7f07128-af06-48f2-b533-db786ae34e64', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.342372ms', 'executionTime': '6.271342ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:08:18,128] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr'
[2022-09-12 11:08:18,154] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee37e50ea445874dfdbbd3a3dd1c7074a3join_yr%27
[2022-09-12 11:08:18,156] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.345635ms
[2022-09-12 11:08:18,157] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '5bd9553b-cea7-4373-994f-baee1ebc68bc', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.345635ms', 'executionTime': '1.282153ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:08:18,256] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:08:18,259] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:18,259] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:18,896] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:18,951] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:08:18,951] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:08:19,053] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:08:19,110] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:08:19,114] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:19,114] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:19,774] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:19,831] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:08:19,832] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:08:19,938] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:08:19,995] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:08:19,995] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:08:19,996] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:08:20,022] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:08:20,049] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:08:20,056] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.594445ms
[2022-09-12 11:08:20,087] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:08:20,114] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:08:20,157] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.044726ms
[2022-09-12 11:08:20,236] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:08:20,236] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 3.767236730818089, 'mem_free': 12865896448, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:08:20,236] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:08:20,242] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:20,242] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:20,876] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:20,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:20,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:21,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:21,453] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:21,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:22,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:22,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:22,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:23,555] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:30,540] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-12 11:08:30,689] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:08:31,147] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:08:31,174] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:08:31,174] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:08:31,230] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:08:31,282] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:08:31,336] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:08:31,337] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:08:31,415] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:08:31,416] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:08:31,442] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:31,572] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:08:31,572] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:31,598] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:08:31,623] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:08:31,623] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:31,649] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:08:31,675] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:08:31,675] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:31,702] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:08:31,728] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:08:31,728] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:31,754] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:08:31,755] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-12 11:08:31,755] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:08:31,756] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 84.716s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_8'}
[2022-09-12 11:08:31,855] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:31,855] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:32,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:32,471] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:08:32,551] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:08:32,551] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-12 11:08:32,552] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:08:32,881] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:08:32,909] - [task:165] INFO -  {'uptime': '680', 'memoryTotal': 15466930176, 'memoryFree': 12996915200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:08:32,934] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:08:32,934] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:08:32,935] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:08:32,965] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:08:33,003] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:08:33,004] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:08:33,035] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:08:33,036] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:08:33,036] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:08:33,036] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:08:33,083] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:08:33,086] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:33,087] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:33,671] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:33,672] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:08:33,792] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:08:33,794] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:08:33,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:33,852] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:33,881] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:08:34,024] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:08:34,053] - [task:165] INFO -  {'uptime': '678', 'memoryTotal': 15466930176, 'memoryFree': 12997033984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:08:34,080] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:08:34,109] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:08:34,109] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:08:34,162] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:08:34,165] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:34,166] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:34,816] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:34,818] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:08:34,944] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:08:34,945] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:08:34,976] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:35,003] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:35,034] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:08:35,162] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:08:35,190] - [task:165] INFO -  {'uptime': '679', 'memoryTotal': 15466930176, 'memoryFree': 12999712768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:08:35,218] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:08:35,248] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:08:35,248] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:08:35,302] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:08:35,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:35,308] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:35,957] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:35,958] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:08:36,084] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:08:36,085] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:08:36,116] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:36,144] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:36,178] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:08:36,303] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:08:36,333] - [task:165] INFO -  {'uptime': '678', 'memoryTotal': 15466930176, 'memoryFree': 12990619648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:08:36,360] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:08:36,388] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:08:36,388] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:08:36,441] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:08:36,445] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:36,445] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:37,086] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:37,087] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:08:37,213] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:08:37,214] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:08:37,247] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:37,273] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:08:37,302] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:08:37,394] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:08:37,915] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:08:42,917] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:08:43,002] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:08:43,010] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:43,013] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:43,603] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:43,604] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:08:43,719] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:08:43,720] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:08:43,720] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:08:44,529] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:08:44,584] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 11:08:44,584] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:08:45,492] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:08:45,837] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:08:46,146] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:08:46,149] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 11:08:46,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:46,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:46,820] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:46,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:46,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:47,729] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:47,737] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:47,737] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:48,840] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:48,851] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:08:48,851] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:08:49,939] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:08:56,581] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:08:56,582] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.3808033573837, 'mem_free': 12997894144, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:08:56,582] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:08:56,582] - [basetestcase:467] INFO - Time to execute basesetup : 24.729475736618042
[2022-09-12 11:08:56,632] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:08:56,632] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:08:56,686] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:08:56,686] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:08:56,739] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:08:56,739] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:08:56,792] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:08:56,792] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:08:56,843] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:08:56,906] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:08:56,907] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:08:56,907] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:09:01,917] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:09:01,921] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:01,921] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:02,541] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:03,597] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:09:03,877] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:09:05,999] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:09:06,081] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:09:06,081] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:09:06,081] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:09:36,112] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:09:36,143] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:09:36,170] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:09:36,242] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 70.091613ms
[2022-09-12 11:09:36,242] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '99ed70bc-21ca-41b8-b02c-dfd5fd58d430', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '70.091613ms', 'executionTime': '69.965367ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:09:36,242] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:09:36,269] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:09:36,294] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:09:36,989] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 692.227908ms
[2022-09-12 11:09:36,989] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:09:37,044] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:09:37,086] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:09:37,095] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.47069ms
[2022-09-12 11:09:37,297] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:09:37,338] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:09:37,360] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:09:37,361] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:09:37,384] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:09:37,386] - [base_gsi:326] INFO - []
[2022-09-12 11:09:38,259] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:09:38,286] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:09:38,355] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.890108ms
[2022-09-12 11:09:38,356] - [base_gsi:282] INFO - BUILD INDEX on default(employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr) USING GSI
[2022-09-12 11:09:39,387] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr) USING GSI
[2022-09-12 11:09:39,411] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%29+USING+GSI
[2022-09-12 11:09:39,439] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 25.403579ms
[2022-09-12 11:09:40,469] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr'
[2022-09-12 11:09:40,494] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%27
[2022-09-12 11:09:40,503] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.798179ms
[2022-09-12 11:09:41,535] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr'
[2022-09-12 11:09:41,562] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%27
[2022-09-12 11:09:41,572] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.295383ms
[2022-09-12 11:09:42,604] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr'
[2022-09-12 11:09:42,631] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%27
[2022-09-12 11:09:42,640] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.668313ms
[2022-09-12 11:09:42,667] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr ON default USING GSI
[2022-09-12 11:09:42,695] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr+ON+default+USING+GSI
[2022-09-12 11:09:42,757] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 59.473805ms
[2022-09-12 11:09:42,794] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr'
[2022-09-12 11:09:42,820] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee3ee17919cbaf4752bb2e9a4a5bf1c955job_title_join_yr%27
[2022-09-12 11:09:42,827] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.920419ms
[2022-09-12 11:09:42,827] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2304ab9d-41e3-4c4a-87c4-521ba8214940', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.920419ms', 'executionTime': '4.84979ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:09:42,929] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:09:42,932] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:42,933] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:43,615] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:43,673] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:09:43,674] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:09:43,780] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:09:43,838] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:09:43,841] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:43,841] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:44,514] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:44,570] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:09:44,571] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:09:44,682] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:09:44,738] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:09:44,738] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:09:44,738] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:09:44,764] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:09:44,790] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:09:44,797] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.084942ms
[2022-09-12 11:09:44,828] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:09:44,860] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:09:44,919] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.831729ms
[2022-09-12 11:09:44,988] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:09:44,988] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 10.28321897560687, 'mem_free': 12847419392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:09:44,988] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:09:44,994] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:44,994] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:45,590] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:45,595] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:45,595] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:46,677] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:46,686] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:46,686] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:47,837] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:47,845] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:47,845] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:48,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:55,459] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-12 11:09:55,604] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:09:56,061] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:09:56,093] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:09:56,093] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:09:56,148] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:09:56,201] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:09:56,253] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:09:56,254] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:09:56,332] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:09:56,333] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:09:56,359] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:56,485] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:09:56,485] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:56,512] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:09:56,537] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:09:56,538] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:56,564] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:09:56,590] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:09:56,590] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:56,618] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:09:56,645] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:09:56,645] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:56,673] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:09:56,673] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-12 11:09:56,673] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:09:56,674] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 84.879s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_9'}
[2022-09-12 11:09:56,771] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:56,772] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:57,452] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:57,486] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:09:57,574] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:09:57,574] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-12 11:09:57,575] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:09:57,799] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:09:57,826] - [task:165] INFO -  {'uptime': '765', 'memoryTotal': 15466930176, 'memoryFree': 12988628992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:09:57,855] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:09:57,855] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:09:57,856] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:09:57,897] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:09:57,935] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:09:57,935] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:09:57,967] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:09:57,968] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:09:57,968] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:09:57,968] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:09:58,018] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:09:58,022] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:58,022] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:58,679] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:58,680] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:09:58,805] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:09:58,806] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:09:58,839] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:09:58,865] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:09:58,896] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:09:59,022] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:09:59,048] - [task:165] INFO -  {'uptime': '763', 'memoryTotal': 15466930176, 'memoryFree': 12987256832, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:09:59,078] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:09:59,105] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:09:59,106] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:09:59,157] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:09:59,160] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:09:59,160] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:09:59,766] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:09:59,767] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:09:59,885] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:09:59,886] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:09:59,919] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:09:59,946] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:09:59,974] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:10:00,095] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:10:00,124] - [task:165] INFO -  {'uptime': '765', 'memoryTotal': 15466930176, 'memoryFree': 12987154432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:10:00,151] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:10:00,181] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:10:00,182] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:10:00,233] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:10:00,240] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:00,241] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:00,987] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:00,988] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:10:01,135] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:10:01,136] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:10:01,165] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:10:01,191] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:10:01,220] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:10:01,338] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:10:01,364] - [task:165] INFO -  {'uptime': '764', 'memoryTotal': 15466930176, 'memoryFree': 12986773504, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:10:01,390] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:10:01,421] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:10:01,421] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:10:01,473] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:10:01,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:01,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:02,159] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:02,160] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:10:02,290] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:10:02,291] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:10:02,324] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:10:02,351] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:10:02,382] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:10:02,476] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:10:03,021] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:10:08,026] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:10:08,113] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:10:08,119] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:08,119] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:08,790] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:08,791] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:10:08,914] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:10:08,915] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:10:08,915] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:10:09,607] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:10:09,664] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:10:09,665] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:10:10,532] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:10:10,837] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:10:11,117] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:10:11,120] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 11:10:11,181] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:11,181] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:11,810] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:11,815] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:11,815] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:12,761] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:12,773] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:12,773] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:13,932] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:13,943] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:13,943] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:15,091] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:21,909] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:10:21,909] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 31.72590261219012, 'mem_free': 13002145792, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:10:21,909] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:10:21,909] - [basetestcase:467] INFO - Time to execute basesetup : 25.140668630599976
[2022-09-12 11:10:21,962] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:10:21,962] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:10:22,019] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:10:22,019] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:10:22,073] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:10:22,073] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:10:22,129] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:10:22,130] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:10:22,184] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:10:22,247] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:10:22,248] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:10:22,248] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:10:27,258] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:10:27,262] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:10:27,262] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:10:27,919] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:10:28,978] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:10:29,269] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:10:31,384] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:10:31,466] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:10:31,466] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:10:31,467] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:11:01,494] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:11:01,523] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:11:01,549] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:11:01,618] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.198051ms
[2022-09-12 11:11:01,618] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd633f7da-8d03-481f-b904-73fcddad5b70', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.198051ms', 'executionTime': '67.124751ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:11:01,618] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:11:01,644] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:11:01,669] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:11:02,372] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 700.441361ms
[2022-09-12 11:11:02,372] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:11:02,430] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:11:02,466] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:11:02,474] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.012296ms
[2022-09-12 11:11:02,694] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:02,733] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:11:02,757] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:11:02,763] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:11:02,783] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:11:02,846] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:03,653] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4cd5d71b9d494a92b824eeab4df44694join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:11:03,679] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4cd5d71b9d494a92b824eeab4df44694join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:11:03,730] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.753123ms
[2022-09-12 11:11:03,730] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4cd5d71b9d494a92b824eeab4df44694join_yr`) USING GSI
[2022-09-12 11:11:04,760] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4cd5d71b9d494a92b824eeab4df44694join_yr`) USING GSI
[2022-09-12 11:11:04,787] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4cd5d71b9d494a92b824eeab4df44694join_yr%60%29+USING+GSI
[2022-09-12 11:11:04,816] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.404138ms
[2022-09-12 11:11:04,856] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4cd5d71b9d494a92b824eeab4df44694join_yr'
[2022-09-12 11:11:04,893] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4cd5d71b9d494a92b824eeab4df44694join_yr%27
[2022-09-12 11:11:04,902] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.250934ms
[2022-09-12 11:11:05,933] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4cd5d71b9d494a92b824eeab4df44694join_yr'
[2022-09-12 11:11:05,958] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4cd5d71b9d494a92b824eeab4df44694join_yr%27
[2022-09-12 11:11:05,967] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.606495ms
[2022-09-12 11:11:06,999] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4cd5d71b9d494a92b824eeab4df44694join_yr'
[2022-09-12 11:11:07,025] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4cd5d71b9d494a92b824eeab4df44694join_yr%27
[2022-09-12 11:11:07,035] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.132669ms
[2022-09-12 11:11:08,066] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:11:08,091] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 11:11:08,095] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.614863ms
[2022-09-12 11:11:08,095] - [base_gsi:504] INFO - {'requestID': '4b56e452-8a20-45b0-924b-73c339e58aec', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4cd5d71b9d494a92b824eeab4df44694join_yr', 'index_id': '215b3e9d1a15b1c7', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.614863ms', 'executionTime': '2.552342ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 11:11:08,096] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:11:08,096] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:11:08,096] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:11:08,097] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:11:08,098] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:11:08,098] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:11:08,099] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:11:08,099] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:11:08,158] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:11:08,195] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:11:08,221] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 11:11:08,398] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 163.470529ms
[2022-09-12 11:11:08,399] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:11:08,399] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:11:10,298] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:10,323] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4cd5d71b9d494a92b824eeab4df44694join_yr'
[2022-09-12 11:11:10,348] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4cd5d71b9d494a92b824eeab4df44694join_yr%27
[2022-09-12 11:11:10,353] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.017874ms
[2022-09-12 11:11:10,354] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1c138d97-902d-4341-859c-6c1c455bd2a0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.017874ms', 'executionTime': '3.951698ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:11:10,452] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:10,454] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:10,454] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:11,213] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:11,270] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:11:11,270] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:11:11,377] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:11:11,439] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:11,442] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:11,443] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:12,105] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:12,161] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:11:12,162] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:11:12,273] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:11:12,330] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:12,330] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:11:12,331] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:11:12,358] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:11:12,385] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:11:12,390] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.543115ms
[2022-09-12 11:11:12,390] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6ef4281c-3721-4959-ac78-8231a60fc863', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '3.543115ms', 'executionTime': '3.483872ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:11:12,447] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:11:12,447] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.46919596112079, 'mem_free': 12825702400, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:11:12,447] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:11:12,451] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:12,451] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:13,144] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:13,149] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:13,149] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:13,828] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:13,833] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:13,833] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:14,814] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:14,825] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:14,825] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:15,975] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:23,624] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-12 11:11:23,766] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:11:23,822] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:11:23,878] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:11:23,937] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:11:23,937] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:11:24,023] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:11:24,025] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:11:24,053] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:24,197] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:11:24,197] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:24,226] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:11:24,254] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:11:24,254] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:24,282] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:11:24,309] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:11:24,310] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:24,339] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:11:24,367] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:11:24,368] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:24,396] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:11:24,396] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-12 11:11:24,396] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:11:24,397] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 87.686s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_10'}
[2022-09-12 11:11:24,487] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:24,487] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:25,181] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:25,214] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:11:25,305] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:11:25,305] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-12 11:11:25,306] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:11:25,514] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:11:25,543] - [task:165] INFO -  {'uptime': '854', 'memoryTotal': 15466930176, 'memoryFree': 12988854272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:11:25,570] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:11:25,571] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:11:25,571] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:11:25,601] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:11:25,639] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:11:25,640] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:11:25,671] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:11:25,672] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:11:25,672] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:11:25,672] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:11:25,725] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:11:25,728] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:25,728] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:26,424] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:26,425] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:11:26,557] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:11:26,559] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:11:26,590] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:26,618] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:26,650] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:11:26,783] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:11:26,810] - [task:165] INFO -  {'uptime': '853', 'memoryTotal': 15466930176, 'memoryFree': 12990119936, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:11:26,836] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:11:26,864] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:11:26,865] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:11:26,917] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:11:26,923] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:26,923] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:27,630] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:27,632] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:11:27,765] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:11:27,766] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:11:27,797] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:27,825] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:27,856] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:11:27,978] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:11:28,006] - [task:165] INFO -  {'uptime': '850', 'memoryTotal': 15466930176, 'memoryFree': 12989984768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:11:28,035] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:11:28,066] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:11:28,066] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:11:28,121] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:11:28,124] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:28,125] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:28,814] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:28,815] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:11:28,948] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:11:28,949] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:11:28,980] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:29,008] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:29,038] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:11:29,168] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:11:29,198] - [task:165] INFO -  {'uptime': '849', 'memoryTotal': 15466930176, 'memoryFree': 12983754752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:11:29,226] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:11:29,258] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:11:29,258] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:11:29,313] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:11:29,316] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:29,316] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:30,004] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:30,005] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:11:30,136] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:11:30,137] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:11:30,170] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:30,197] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:11:30,226] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:11:30,327] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:11:30,859] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:11:35,864] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:11:35,953] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:11:35,957] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:35,957] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:36,645] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:36,646] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:11:36,776] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:11:36,777] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:11:36,778] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:11:37,444] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:11:37,511] - [on_prem_rest_client:3047] INFO - 0.07 seconds to create bucket default
[2022-09-12 11:11:37,512] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:11:38,431] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:11:38,748] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:11:39,043] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:11:39,045] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 11:11:39,109] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:39,110] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:39,812] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:39,818] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:39,818] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:40,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:40,875] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:40,875] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:42,052] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:42,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:42,062] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:43,233] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:50,102] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:11:50,102] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 34.95102602102802, 'mem_free': 12983918592, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:11:50,103] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:11:50,103] - [basetestcase:467] INFO - Time to execute basesetup : 25.618372201919556
[2022-09-12 11:11:50,155] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:11:50,155] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:11:50,209] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:11:50,210] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:11:50,266] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:11:50,266] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:11:50,324] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:11:50,325] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:11:50,382] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:11:50,445] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:11:50,447] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:11:50,451] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:11:55,463] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:11:55,466] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:11:55,466] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:11:56,092] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:11:57,166] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:11:57,345] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:12:00,515] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:12:00,592] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:00,593] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:12:00,593] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:12:30,620] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:12:30,650] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:12:30,675] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:12:30,743] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 65.79964ms
[2022-09-12 11:12:30,743] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8c116719-6ced-4ad1-a8a3-bc3810c7abcd', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '65.79964ms', 'executionTime': '65.732186ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:12:30,743] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:12:30,769] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:12:30,794] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:12:31,471] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 675.788627ms
[2022-09-12 11:12:31,471] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:12:31,516] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:12:31,572] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:12:31,582] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.759747ms
[2022-09-12 11:12:31,812] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:31,851] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:12:31,865] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:12:31,866] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:12:31,882] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:12:31,958] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:32,773] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeedca659164109444ab00eea4420ea46f8join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:12:32,798] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeedca659164109444ab00eea4420ea46f8join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:12:32,862] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 60.884295ms
[2022-09-12 11:12:32,863] - [base_gsi:282] INFO - BUILD INDEX on default(`employeedca659164109444ab00eea4420ea46f8join_yr`) USING GSI
[2022-09-12 11:12:33,893] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeedca659164109444ab00eea4420ea46f8join_yr`) USING GSI
[2022-09-12 11:12:33,919] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeedca659164109444ab00eea4420ea46f8join_yr%60%29+USING+GSI
[2022-09-12 11:12:33,953] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.327954ms
[2022-09-12 11:12:34,008] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedca659164109444ab00eea4420ea46f8join_yr'
[2022-09-12 11:12:34,041] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedca659164109444ab00eea4420ea46f8join_yr%27
[2022-09-12 11:12:34,050] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.026733ms
[2022-09-12 11:12:35,082] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedca659164109444ab00eea4420ea46f8join_yr'
[2022-09-12 11:12:35,109] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedca659164109444ab00eea4420ea46f8join_yr%27
[2022-09-12 11:12:35,117] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.166708ms
[2022-09-12 11:12:36,148] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedca659164109444ab00eea4420ea46f8join_yr'
[2022-09-12 11:12:36,173] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedca659164109444ab00eea4420ea46f8join_yr%27
[2022-09-12 11:12:36,180] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.721002ms
[2022-09-12 11:12:37,212] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:12:37,237] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 11:12:37,242] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.717626ms
[2022-09-12 11:12:37,242] - [base_gsi:504] INFO - {'requestID': 'faa9d6bf-4feb-4bd2-997d-7f4f66a6e0d1', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeedca659164109444ab00eea4420ea46f8join_yr', 'index_id': '379d8ef1eb8849e5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.717626ms', 'executionTime': '2.646415ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 11:12:37,242] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:12:37,243] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:37,243] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:12:37,243] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:12:37,243] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:37,244] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:37,244] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:37,244] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:37,303] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:37,341] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:12:37,366] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 11:12:37,561] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 182.820367ms
[2022-09-12 11:12:37,561] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:12:37,562] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:12:38,765] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-12 11:12:38,820] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-12 11:12:38,848] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:12:38,874] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 11:12:38,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.663288ms
[2022-09-12 11:12:38,877] - [base_gsi:504] INFO - {'requestID': '43bc081d-4bde-40ee-aafc-ba456d198e3b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeedca659164109444ab00eea4420ea46f8join_yr', 'index_id': '379d8ef1eb8849e5', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.663288ms', 'executionTime': '1.602321ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 11:12:38,878] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:12:38,878] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:38,878] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:12:38,878] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:12:38,879] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:38,879] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:38,879] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:38,879] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:12:38,932] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:12:38,969] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:12:38,994] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 11:12:39,100] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 96.645309ms
[2022-09-12 11:12:39,100] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:12:39,100] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:12:40,321] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeedca659164109444ab00eea4420ea46f8join_yr ON default USING GSI
[2022-09-12 11:12:40,349] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeedca659164109444ab00eea4420ea46f8join_yr+ON+default+USING+GSI
[2022-09-12 11:12:40,393] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 41.912311ms
[2022-09-12 11:12:40,430] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeedca659164109444ab00eea4420ea46f8join_yr'
[2022-09-12 11:12:40,457] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeedca659164109444ab00eea4420ea46f8join_yr%27
[2022-09-12 11:12:40,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.232431ms
[2022-09-12 11:12:40,465] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '530c0df8-a98f-4019-b764-9828d08bc8d0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '6.232431ms', 'executionTime': '6.162791ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:12:40,571] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:40,575] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:40,576] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:41,259] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:41,312] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:12:41,313] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:12:41,424] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:12:41,482] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:41,485] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:41,485] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:42,202] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:42,257] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:12:42,257] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:12:42,375] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:12:42,429] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:12:42,430] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:12:42,430] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:12:42,458] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:12:42,484] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:12:42,493] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.408046ms
[2022-09-12 11:12:42,524] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:12:42,578] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:12:42,634] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.651692ms
[2022-09-12 11:12:42,747] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:12:42,748] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.61574470148989, 'mem_free': 12822261760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:12:42,748] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:12:42,752] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:42,753] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:43,465] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:43,469] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:43,470] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:44,183] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:44,187] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:44,188] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:45,341] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:45,348] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:45,349] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:46,547] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:54,121] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-12 11:12:54,264] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:12:55,014] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:12:55,044] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:12:55,044] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:12:55,102] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:12:55,156] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:12:55,209] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:12:55,209] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:12:55,287] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:12:55,288] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:12:55,313] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:55,463] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:12:55,463] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:55,492] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:12:55,520] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:12:55,520] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:55,546] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:12:55,571] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:12:55,571] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:55,599] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:12:55,626] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:12:55,626] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:55,658] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:12:55,658] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-12 11:12:55,658] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:12:55,659] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 91.228s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=plasma -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'plasma', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_11'}
[2022-09-12 11:12:55,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:55,744] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:56,439] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:56,471] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:12:56,551] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:12:56,552] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-12 11:12:56,552] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:12:56,770] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:12:56,797] - [task:165] INFO -  {'uptime': '944', 'memoryTotal': 15466930176, 'memoryFree': 12967542784, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:12:56,823] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:12:56,823] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:12:56,823] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:12:56,863] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:12:56,899] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:12:56,899] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:12:56,927] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:12:56,928] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:12:56,928] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:12:56,929] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:12:56,977] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:12:56,980] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:56,981] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:57,655] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:57,656] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:12:57,781] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:12:57,782] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:12:57,812] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:12:57,838] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:12:57,867] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:12:57,988] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:12:58,014] - [task:165] INFO -  {'uptime': '944', 'memoryTotal': 15466930176, 'memoryFree': 12969820160, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:12:58,040] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:12:58,068] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:12:58,068] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:12:58,119] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:12:58,122] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:58,122] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:12:58,830] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:12:58,832] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:12:58,958] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:12:58,959] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:12:58,991] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:12:59,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:12:59,047] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:12:59,170] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:12:59,198] - [task:165] INFO -  {'uptime': '941', 'memoryTotal': 15466930176, 'memoryFree': 12968960000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:12:59,226] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:12:59,255] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:12:59,256] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:12:59,309] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:12:59,314] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:12:59,314] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:00,006] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:00,007] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:13:00,144] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:13:00,145] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:13:00,176] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:13:00,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:13:00,233] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:13:00,355] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:13:00,383] - [task:165] INFO -  {'uptime': '939', 'memoryTotal': 15466930176, 'memoryFree': 12969930752, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:13:00,410] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:13:00,438] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:13:00,439] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:13:00,490] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:13:00,493] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:00,493] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:01,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:01,163] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:13:01,290] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:13:01,292] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:13:01,321] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:13:01,347] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:13:01,375] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:13:01,464] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:13:02,012] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:13:07,017] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:13:07,108] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:13:07,113] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:07,113] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:07,818] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:07,820] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:13:07,953] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:13:07,954] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:13:07,954] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:13:08,624] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:13:08,685] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:13:08,686] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:13:09,625] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:13:09,957] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:13:10,240] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:13:10,242] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 11:13:10,300] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:10,300] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:10,997] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:11,002] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:11,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:12,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:12,047] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:12,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:13,300] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:13,319] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:13,319] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:14,559] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:21,617] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:13:21,618] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 33.82194881388777, 'mem_free': 12968763392, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:13:21,618] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:13:21,618] - [basetestcase:467] INFO - Time to execute basesetup : 25.87742280960083
[2022-09-12 11:13:21,670] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:13:21,670] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:13:21,729] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:13:21,729] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:13:21,782] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:13:21,782] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:13:21,835] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:13:21,835] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:13:21,886] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:13:21,954] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:13:21,955] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:13:21,966] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:13:26,981] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:13:26,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:13:26,986] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:13:27,673] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:13:28,740] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:13:29,049] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:13:32,025] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:13:32,107] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:13:32,107] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:13:32,107] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:14:02,135] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:14:02,165] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:14:02,190] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:14:02,259] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 67.765409ms
[2022-09-12 11:14:02,260] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'c1f9e420-752d-4fd4-8654-b26b48fbcca7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '67.765409ms', 'executionTime': '67.704224ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:14:02,260] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:14:02,285] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:14:02,310] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:14:03,010] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 698.405754ms
[2022-09-12 11:14:03,010] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:14:03,074] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:14:03,109] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:14:03,118] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.663546ms
[2022-09-12 11:14:03,317] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:03,354] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:14:03,371] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:14:03,372] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:14:03,388] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:14:03,472] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:04,277] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef59f434d261e4c9aa1c67ce4b0635e79join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-12 11:14:04,305] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-12 11:14:04,405] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 98.167759ms
[2022-09-12 11:14:04,406] - [base_gsi:282] INFO - BUILD INDEX on default(`employeef59f434d261e4c9aa1c67ce4b0635e79join_yr`) USING GSI
[2022-09-12 11:14:05,436] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeef59f434d261e4c9aa1c67ce4b0635e79join_yr`) USING GSI
[2022-09-12 11:14:05,462] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%60%29+USING+GSI
[2022-09-12 11:14:05,484] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.389519ms
[2022-09-12 11:14:05,535] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr'
[2022-09-12 11:14:05,568] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%27
[2022-09-12 11:14:05,577] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.594387ms
[2022-09-12 11:14:06,606] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr'
[2022-09-12 11:14:06,633] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%27
[2022-09-12 11:14:06,643] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.158559ms
[2022-09-12 11:14:07,675] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr'
[2022-09-12 11:14:07,703] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%27
[2022-09-12 11:14:07,712] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.978285ms
[2022-09-12 11:14:08,744] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:14:08,771] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-12 11:14:08,775] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.630833ms
[2022-09-12 11:14:08,775] - [base_gsi:504] INFO - {'requestID': '8736e060-7101-4ada-99b4-f562f0b7dba6', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr', 'index_id': '352462db2ac3642e', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.630833ms', 'executionTime': '2.558679ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-12 11:14:08,776] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-12 11:14:08,776] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:14:08,776] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-12 11:14:08,777] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-12 11:14:08,777] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:14:08,777] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:14:08,778] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:14:08,778] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-12 11:14:08,838] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-12 11:14:08,878] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-12 11:14:08,904] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-12 11:14:09,089] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 171.106373ms
[2022-09-12 11:14:09,090] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-12 11:14:09,090] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-12 11:14:11,245] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 11:14:13,782] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:14:13,841] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-12 11:14:13,841] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:14:14,789] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:14:15,114] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:14:15,419] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:14:15,421] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-12 11:14:17,458] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:14:17,459] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:17,488] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:14:17,519] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:14:17,519] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:17,548] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:14:17,578] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:14:17,579] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:17,607] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:14:17,636] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:14:17,637] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:17,665] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:14:17,716] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:17,742] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr'
[2022-09-12 11:14:17,768] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%27
[2022-09-12 11:14:17,813] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.780958ms
[2022-09-12 11:14:17,813] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '42c1b1cc-6d24-466f-8c67-8f059f246d55', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '42.780958ms', 'executionTime': '42.711391ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:14:17,839] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef59f434d261e4c9aa1c67ce4b0635e79join_yr'
[2022-09-12 11:14:17,865] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef59f434d261e4c9aa1c67ce4b0635e79join_yr%27
[2022-09-12 11:14:17,868] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.285767ms
[2022-09-12 11:14:17,868] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '3b654c50-fcf9-4112-af2b-a309731aa094', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.285767ms', 'executionTime': '1.225317ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:14:17,922] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:18,061] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:18,064] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:18,064] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:18,826] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:18,891] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:14:18,891] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:14:19,012] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:14:19,070] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:19,073] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:19,073] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:19,806] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:19,863] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:14:19,864] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:14:19,979] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:14:20,038] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:14:20,038] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:14:20,038] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:14:20,063] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:14:20,088] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:14:20,097] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.30591ms
[2022-09-12 11:14:20,097] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '33e8ed76-224d-4cb3-8c8b-a590cf439b21', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '7.30591ms', 'executionTime': '7.227385ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:14:20,098] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:14:20,124] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:14:20,150] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:14:20,153] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.291332ms
[2022-09-12 11:14:20,153] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1d069222-ce99-475e-bcdc-ed20be938911', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.291332ms', 'executionTime': '1.228702ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:14:20,204] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:14:20,204] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.57757535876424, 'mem_free': 12782125056, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:14:20,204] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:14:20,207] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:20,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:20,926] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:20,931] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:20,931] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:21,818] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:21,826] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:21,827] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:23,083] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:23,091] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:23,091] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:24,378] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:32,075] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 11:14:32,215] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:14:32,528] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:14:32,557] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:14:32,557] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:14:32,614] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:14:32,672] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:14:32,728] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:14:32,729] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:14:32,815] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:14:32,816] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:14:32,841] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:32,978] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:14:32,978] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:33,009] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:14:33,038] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:14:33,038] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:33,066] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:14:33,094] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:14:33,094] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:33,122] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:14:33,151] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:14:33,151] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:33,180] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:14:33,180] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-12 11:14:33,180] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:14:33,181] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 97.496s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-12_10-57-08/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-12 11:14:33,243] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:33,244] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:34,008] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:34,044] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:14:34,130] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1969-rel-EE-enterprise
[2022-09-12 11:14:34,130] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-12 11:14:34,131] - [basetestcase:224] INFO - initializing cluster
[2022-09-12 11:14:34,270] - [task:160] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-12 11:14:34,298] - [task:165] INFO -  {'uptime': '1041', 'memoryTotal': 15466930176, 'memoryFree': 12951826432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:14:34,325] - [task:204] INFO - quota for index service will be 256 MB
[2022-09-12 11:14:34,325] - [task:206] INFO - set index quota to node 127.0.0.1 
[2022-09-12 11:14:34,325] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-12 11:14:34,363] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-12 11:14:34,394] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-12 11:14:34,395] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-12 11:14:34,424] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-12 11:14:34,425] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-12 11:14:34,425] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-12 11:14:34,425] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-12 11:14:34,473] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:14:34,476] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:34,476] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:35,167] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:35,168] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:14:35,293] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:14:35,294] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:14:35,324] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:35,350] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:35,378] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:14:35,500] - [task:160] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-12 11:14:35,527] - [task:165] INFO -  {'uptime': '1039', 'memoryTotal': 15466930176, 'memoryFree': 12946608128, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:14:35,552] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:14:35,579] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-12 11:14:35,579] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-12 11:14:35,629] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:14:35,632] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:35,632] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:36,309] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:36,311] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:14:36,443] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:14:36,445] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:14:36,475] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:36,500] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:36,532] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:14:36,650] - [task:160] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-12 11:14:36,681] - [task:165] INFO -  {'uptime': '1041', 'memoryTotal': 15466930176, 'memoryFree': 12941950976, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:14:36,708] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:14:36,739] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-12 11:14:36,740] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-12 11:14:36,792] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:14:36,795] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:36,795] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:37,560] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:37,561] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:14:37,699] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:14:37,700] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:14:37,730] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:37,757] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:37,786] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:14:37,905] - [task:160] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-12 11:14:37,932] - [task:165] INFO -  {'uptime': '1040', 'memoryTotal': 15466930176, 'memoryFree': 12950638592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1969-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-12 11:14:37,958] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-12 11:14:37,984] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-12 11:14:37,985] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-12 11:14:38,037] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-12 11:14:38,040] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:38,040] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:38,779] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:38,780] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:14:38,915] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:14:38,916] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:14:38,946] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:38,971] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-12 11:14:38,999] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=plasma
[2022-09-12 11:14:39,086] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-12 11:14:39,591] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:14:44,595] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-12 11:14:44,679] - [basetestcase:262] INFO - done initializing cluster
[2022-09-12 11:14:44,683] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:44,683] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:45,398] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:45,399] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-12 11:14:45,526] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-12 11:14:45,526] - [remote_util:5249] INFO - b'ok'
[2022-09-12 11:14:45,527] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-12 11:14:46,201] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-12 11:14:46,256] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-12 11:14:46,256] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-12 11:14:47,171] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:14:47,499] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:14:47,790] - [task:383] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-12 11:14:47,795] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-12 11:14:47,858] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:47,858] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:48,645] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:48,650] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:48,651] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:49,828] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:49,840] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:49,840] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:51,137] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:51,150] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:14:51,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:14:52,438] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:14:59,843] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:14:59,843] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 35.15997910594524, 'mem_free': 12947369984, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:14:59,843] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:14:59,843] - [basetestcase:467] INFO - Time to execute basesetup : 26.603169202804565
[2022-09-12 11:14:59,900] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:14:59,901] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:14:59,960] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:14:59,961] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:15:00,017] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:15:00,018] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:15:00,076] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-12 11:15:00,076] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-12 11:15:00,129] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:00,189] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'plasma'} set
[2022-09-12 11:15:00,190] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-12 11:15:00,190] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-12 11:15:05,201] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-12 11:15:05,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:05,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:05,931] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:07,129] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-12 11:15:07,310] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-12 11:15:10,217] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-12 11:15:10,295] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:10,295] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-12 11:15:10,295] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-12 11:15:40,326] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-12 11:15:40,356] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:15:40,381] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:15:40,449] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 66.428602ms
[2022-09-12 11:15:40,450] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '08aea203-fed3-4e0d-8240-dca650e9ea09', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '66.428602ms', 'executionTime': '66.358714ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-12 11:15:40,450] - [tuq_helper:758] INFO - Create primary index
[2022-09-12 11:15:40,476] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-12 11:15:40,501] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-12 11:15:41,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 669.681229ms
[2022-09-12 11:15:41,173] - [tuq_helper:760] INFO - Check if index is online
[2022-09-12 11:15:41,228] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:15:41,261] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:15:41,268] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.904044ms
[2022-09-12 11:15:41,469] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:41,503] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-12 11:15:41,523] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-12 11:15:41,524] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-12 11:15:41,544] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-12 11:15:41,615] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:41,740] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:41,743] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:41,743] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:42,504] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:42,558] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:15:42,559] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-12 11:15:42,684] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:15:42,742] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:42,746] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:42,746] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:43,541] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:43,598] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-12 11:15:43,598] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-12 11:15:43,725] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-12 11:15:43,783] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-12 11:15:43,783] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-12 11:15:43,784] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:15:43,810] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-12 11:15:43,837] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-12 11:15:43,846] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.419751ms
[2022-09-12 11:15:43,872] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-12 11:15:43,898] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-12 11:15:43,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 55.733977ms
[2022-09-12 11:15:44,026] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-12 11:15:44,027] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.153769569146526, 'mem_free': 12773699584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-12 11:15:44,027] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-12 11:15:44,031] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:44,031] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:44,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:44,827] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:44,828] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:46,117] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:46,126] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:46,126] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:47,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:47,450] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-12 11:15:47,451] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-12 11:15:48,832] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-12 11:15:56,568] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-12 11:15:56,720] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-12 11:15:57,742] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-12 11:15:57,770] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-12 11:15:57,770] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-12 11:15:57,828] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:15:57,897] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:15:57,955] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-12 11:15:57,956] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-12 11:15:58,042] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-12 11:15:58,043] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-12 11:15:58,070] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:15:58,212] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-12 11:15:58,212] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:15:58,240] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-12 11:15:58,267] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-12 11:15:58,267] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:15:58,296] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-12 11:15:58,323] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-12 11:15:58,323] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:15:58,351] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-12 11:15:58,377] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-12 11:15:58,377] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-12 11:15:58,406] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-12 11:15:58,406] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-12 11:15:58,406] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-12 11:15:58,407] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 85.247s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 11415 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Succeeded


Version: versions-12.09.2022-09.31.cfg
Build Log: make-12.09.2022-09.31.log
Server Log: logs-12.09.2022-09.31.tar.gz

Finished