Building

Started building at 2022/09/02 01:46:26
Using pegged server, 1948 build
Calculating base
Updating mirror
Basing run on 7.2.0-1948 3df08568d
Updating tree for run 02.09.2022-01.46
query is at c3f3821, changes since last good build: 
 c3f3821ea MB-53533 Ensure initialisation is complete ahead of PrepareTopologyChange processing.
 e74608abe MB-53541 Independent exchange notifier for merge actions.
 6a1b677ce MB-53565 Do not consider primary index for index scan inside correlated subquery
 89f2c4277 MB-53565 Proper formalization for ExpressionTerm and SubqueryTerm under ANSI join
 3c0a17984 MB-53565 Do not use join predicates as index span if under hash join
 f636e8c63 MB-53506 Additional fixes for unit tests
 b5c561c25 MB-53506 Fix some unit test issues
 6de332853 MB-53230. Change load_factor to moving avg
 8fcba5ab8 MB-53410 Revise fix.
 5ce5fd088 MB-53528 add SkipMetering to index connection
 8047faf2e MB-53526 register unbillable admin units with regulator
 cae373174 MB-53410 Reduce redundant value creation in UNNEST.
 be0672272 MB-52176:If use_cbo setting is true then run update statistics after CREATE INDEX for non deferred indexes and after BUILD INDEX for deferred indexes
 d0e4f965a updating go mod for building in module aware mode
 f3f2f3580 MB-53514 Revise fix.
 08344c66a MB-53514 Report KV RUs incurred for sequential scans
 120933d52 Revert "MB-53514 Report KV RUs incurred for sequential scans"
 f70f23c4f MB-53514 Report KV RUs incurred for sequential scans
 63fbf9734 MB-53406 Reduce atomics in lockless pool.
 c68ea6550 MB-53506 Prevent multiple mutations on a single key in UPSERT.
 407a2027f MB-53477 Alter threshold for spilling group-by without a quota.
 b7552a36e MB-53472:Add command line option -query_context or -qc to specify the query_context when connecting to the couchbase cluster to launch cbq shell
 1613c282d MB-53506 Improve sequential scan integration with IUD operations.
 c754bf3c9 MB-53176:Perform memory accounting when cached values are freed in the ExpressionScan operator
 e178f11e6 MB-33748 Go fmt
 174adf156 MB-33748 Making time in catalogs ISO-8601 format
 73e10feeb MB-53230. Use minimum value
 1c61edc90 MB-51928 Additional fixes for join filter
 839c44620 MB-44219 credentials not always available to planner
 1c9c1c6a1 MB-53439 MB-50997 scope UDF definitions stored in _system scope, open system:functions
 456304363 MB-53446 Honor USE NL hint for kv range scan
 d1fc29ec6 MB-53444 Avoid invalid numbers when calculating group cost
 f2d314563 MB-44219: In serverless mode, check if user has permissions on bucket when executing a scoped function
 b72782619 MB-53434. return false only on error
 519566933 go mod tidy (update to bleve@v2.3.4)
 c327709fe MB-52098: go mod tidy
 ab8a702f2 MB-53427 cbq vim-mode: enhance commands
 a85b778e2 MB-53420 consider empty non nil parameter list for inline UDF caching
 568270f72 MB-53416. Set Conditional for NVL/NVL2/Decode/IfMissing/IfNull
 6502af299 MB-53391 Fix value tracking for RAW projections
 d1e41aa49 MB-29604 Fix test cases.
 0b269607c MB-44305 Spill sort and group by to disk.
 e8b2e1f49 MB-53394 Don't escape HTML in errors and warnings.
 8d2e43a17 MB-29604 Return warning on division by zero.
 6a6d16cb5 MB-53372 properly formalize select in UDF with variables as non correlated, avoid caching for UDF select
 0faaadf0c MB-53377 Proper set up of join keys when converting from ANSI join to lookup join
 f47e3d29b MB-53371 Use 64-bit value for 'Uses'
 86c6a347a MB-53353 Adjust unbounded folllowing window range
 039b1d27c GOCBC-1325 Add Transaction logger
 c4f13bfb9 MB-52952 - [FTS] Improving the precision of stats used for WUs metering
 4d2617f4a MB-44219 In serverless mode if the user is not an admin user and tries to access a collection, scope or bucket that does not exist or they do not have permissions on:  If the user has permissions on the bucket in the keyspace they are trying to access, display the specific error message. If not, display the generic access denied message.
 758bc9411 MB-52176 Perform automatic UPDATE STATISTICS for CREATE PRIMARY INDEX
 38fa22e85 MB-53254 Ignore transaction binary documents in sequential scans
 5dd58d359 MB-53291 Administrator is throttled
 5fcf49bd5 MB-53298 Do not return error for MISSING value in join processing
 8b1553f13 MB-53248 sub doc api to account for rus
 34267f151 MB-52971 Handing CheckResultReject
 0d0d028ec MB-53260 Fix last_scan_time for sequential scans.
 70e3a059c -= passing TlsCAFile to regulator init (unused/deprecated)
 c32b4fb19 "MB-53230 load factor of query node"
 28049ab09 MB-51961 Fix typo
 47cd8a06a Build failure Revert "MB-53230 load factor of query node"
 46c894124 Revert "MB-44305 Spill sort and group by to disk."
 9f11a3e8b MB-51961 Allow primary index to be used on inner of nested-loop join
 95839d370 MB-44305 Spill sort and group by to disk.
 4d1078dd1 MB-53230 load factor of query node
 422c73fe7 MB-53235 leave item quota management to producer for serialized operators
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 90035b0, changes since last good build: 
 90035b0e7 MB-52195 Tag "system" collections to not be metered
 dd4f8b6de MB-52142: Fix default for bucket throttle setting
 f5a520077 MB-50537: Don't allow strict n2n encryption if external unencrypted dist
 823c23c4d MB-53478: Fix saving anonymous functions to disk
 badb66c8f MB-52142: Add throttle limits to bucket config
 f09932bb4 MB-52044 Fix eaccess crash
 f2b6fed41 MB-52142: Add throttle limits to internal settings
 2efa104ee MB-52350 Allow setting per-bucket storage limits
 7e4c5811d Prevent memcached bucket creation on serverless
 28db38600 MB-52350 Fix default values for storage limits
 8fff21898 MB-51516: Don't clamp in_docs estimate to current checkpoint items
 c2a03f4bf MB-52226: Introduce pause/resume APIs that are stubbed out
 9cffcd45e MB-23768: Don't allow user backup in mixed clusters
 f3e9ba2f0 MB-23768: Fix validator:json_array()
 9e90c3142 MB-23768: Add menelaus_users:store_users and use it for restore
 1f3a00032 MB-23768: Add replicated_dets:change_multiple/2
 0c909be9c MB-23768: [rbac] Make sure we compress docs only once when ...
 a804ff225 MB-23768: Add PUT /settings/rbac/backup
 85a2b99a7 MB-23768: Move security_roles_access and ldap_access checks...
 807549ea8 MB-23768: Fix validator:apply_multi_params
 c85963d04 MB-23768: Add GET /settings/rbac/backup
 253346b7b MB-23768: Call menelaus_roles:validate_roles when validating...
 7b4f6c40e MB-23768: Remove unnecessary has_permission(Permission, Req) check
 8301f1565 MB-23768: Remove permission param in verify_ldap_access
 b08f974b7 MB-53326 Push CCCP payload on all kv nodes
 19898a852 MB-52350 Fix unused variable
 666af3431 MB-52350 Add storage limits to bucket config
 8ae0cbb85 MB-52350 Add storage limits to internal settings
 6b01ad610 MB-53423 Adjust bucket maximums for _system scope
 44aa2ee1f MB-53288: New query node-quota parameter
 207277058 MB-53352 Report the running config profile
 65faa5fe7 MB-51738 Use this_node() in ns_memcached
 3ab091f2e MB-51738 Define this_node() to handle distribution crash
 f9693814a MB-53192: Add upgrade for memory alerts
 763b16746 MB-53193: Reenable autofailover popup alerts
 97fce2439 MB-47905: Pass client cert path to services
 e9572c7c9 Update regulator frequently_changed_key prefix to /regulator/report
 55725498f MB-53323: consider keep nodes when placing buckets in rebalance
 492ae395e Add isServerless to /pools result
couchstore is at 803ec5f, changes since last good build: 
 803ec5f Refactor: mcbp::datatype moved to cb::mcbp::datatype
forestdb is at acba458, changes since last good build: none
kv_engine is at 1d85f5a, changes since last good build: 
 1d85f5a55 MB-53127: Document write should clear read usage
 d69988fa9 MB-52311: [1/n] Pause / Resume Bucket: opcodes
 225d4a7ea Refactor bucket delete to add extra extra unit tests
 c14da3f90 MB-53510: Refactor bucket creation
 8a3da42c2 MB-53543: Disable BackfillSmallBuffer test
 cb7a5b432 MB-53304: Enforce holding of stateLock in VBucket::queueItem [2/3]
 e74ba64e4 MB-53304: Enforce holding of stateLock in VBucket::queueItem [1/3]
 0ceebade0 Remove ServerCallbackIface
 19d61765d MB-52553: Don't special-case persistence cursor in CM::addStats
 5836d9a70 MB-50984: Remove max_checkpoints hard limit on the single vbucket
 47572e321 MB-50984: Default checkpoint_destruction_tasks=2
 00201d310 MB-53523: Only check snap start vs last snap end if active VB
 c90a937a4 Reformat test_reader_thread_starvation_warmup
 dec1c6c2c Merge "Merge branch 'neo' into 'master'"
 af74c95fe Refactor: CheckpointManager::registerCursorBySeqno()
 a933cf568 MB-53448: DCP_ADD_STREAM_FLAG_TO_LATEST should use highSeqno of collections(s) in filter
 08438bfb8 MB-53259: Update DCP Consumer byffer-size at dynamic Bucket Quota change
 e61d09645 [Refactor] deselect bucket before trying to delete
 afac71aab MB-53055: Fix Checkpoint::isEmptyByExpel() semantic
 ae8baf2dc Remove unused code from kvstore_test
 6f5ba689c [Refactor] Move bufferevent related code to subclass
 2dd1745c6 MB-53498: Delay bucket type update
 d50b99685 Merge branch 'neo' into 'master'
 79de292f2 Merge branch 'neo' into 'master'
 34bc1c7d2 Merge "Merge branch 'neo' into 'master'"
 074db327f Enable KVStoreTest GetBySeqno for non-couchstore
 16bd96ae6 MB-53284: Use magma memory optimized writes in BucketQuotaChangeTest
 854eced08 Merge branch 'neo' into 'master'
 50f5747b7 Merge "Merge branch 'neo' into 'master'"
 c231af910 Cleanup: remove 'polling' durability timeout mode
 f5930b3ea Tidy: Checkpoint::queueDirty use structured binding in for loop
 a65ca2ba1 Merge branch 'neo' into 'master'
 16f186be2 Only regenerate serverless/configuration.json if exe changed
 194900077 MB-53052: Remove KVBucket::itemFreqDecayerIsSnoozed()
 4b4ad639d Refactor: Create factory method for Connection objects
 f3ac46848 MB-35297: Fix RangeScan sampling stats NotFound path
 7d3f297f7 MB-46738: Rename dcp_conn_buffer_ratio into dcp_consumer_buffer_ratio
 9bc866891 [Refactor] Remove the history field of sloppy gauge
 a7b78c756 MB-53055: Add highestExpelledSeqno to Checkpoint ostream
 a58bd636c MB-53055: Add highest_expelled_seqno to Checkpoint stats
 7e4587d1e Remove duplicate method in DurabilityEPBucketTest
 12136509b Add labels to Montonic<> members of Checkpoint
 795dd8dc0 MB-53055: Fix exception message in CM::registerCursorBySeqno
 200aa87ae Add "filter" capabilities to delete bucket
 6fcfed646 SetClusterConfig should create config-only bucket
 6990718c8 MB-52953: Remove refs to old replication-throttle params and stats
 0fdcf8882 MB-52953: Remove unused EPStats::replicationThrottleThreshold
 bc4592d8b MB-52953: Use mutation_mem_threshold in ReplicationThrottleEP::hasSomeMemory
 b1ed0feb2 MB-52953: Turn mutation_mem_threshold into mutation_mem_ratio
 45dd2db60 MB-53429: Hold vbState lock during pageOut
 ba18b10ca MB-53438: Acquire the vbState lock during disk backfill
 348287953 MB-53141: Return all if sampling range-scan requests samples > keys
 9da38ff86 MB-35297: Improve logging for RangeScan create/cancel
 79aa3dd72 MB-53100: Add extra seqno log information after we register a cursor
 415b3ec74 MB-53198: Do not abort warmup for shard if scan cancelled
 dc09bb535 Cleanup: Move mcbp::datatype to cb::mcbp::datatype
 a77fca118 MB-35297: Meter RangeScan create
 36d090abe MB-35297: Throttle RangeScan create/continue
 40321cf27 SetClusterConfig should handle all bucket states
 ac0c0486d Merge commit 'couchbase/neo~7' into trunk
 c615d15f2 Merge "Merge commit 'couchbase/neo~10' into trunk"
 a96e4a5e9 MB-52806: Disconnect DCP connections when they loose privilege
 6b7d68b4e MB-52158: Check for privilege in RangeScan continue/cancel
 b887f1f17 Merge commit 'couchbase/neo~10' into trunk
 2cfe963a7 Modernize config parsing [2/2]
 4a6018627 MB-53359: Add uniqe error code for config-bucket
 e0e5d5c98 MB-35297: Add EventDrivenTimeoutTask
 8bfdba483 Cleanup: move mcbp::subdoc under cb::mcbp::subdoc
 cf97e6792 Cleanup: Move mcbp::cas under cb::mcbp::cas
 3834eb115 MB-43127: Log succcess status from dumpCallback
 bcb730456 MB-52172 Refactor source file generation cmake target
 d847f8a55 MB-35297: Meter RangeScan key/values
 a7a610b48 Refactor: Rename CreateBucketCommandContext
 af47290a6 Refactor out wait code to separate method
 3c30a1142 Include all bucket states in "bucket_details "
 5d272f547 MB-53379: Allow Collection enabled clients to select COB
 0042495b9 MB-52975: Fold backfill create and scan into one invocation of run
 53f915d1d MB-35297: runtime must not be zero when backfill completes
 a811f317b MB-53359: Don't try to fetch bucket metrics from config-only bucket
 881774c5e MB-53354: Extend CheckpointMemoryTrackingTest suite for non-SSO case
 7d7389df7 Modernize parse_config [1/2]
 72e650860 Set the correct hostname for dcp metering test
 8325ff14b Remove support for DT_CONFIGFILE
 92c8f4fa8 Remove config_parse from server-api
 f85f41bad MB-35297: RangeScan document 'flags' should match GetMeta byte order
 3eccd2aa6 MB-53157: RangeScanCreate uuid should be a string
 67d4759c0 MB-52953: Add ReplicationThrottleEP::engine member
 c310b2f4a Don't use the term whitelist
 407905037 MB-53197: Add support for ClusterConfigOnly bucket
 f61b2e1c6 MB-53294: Introduce storage_bytes metering metric
 be1577087 MB-52953: Remove unused UseActiveVBMemThreshold
 ecbd40992 MB-35297: Add missing recvResponse / sendCommand from RangeScanTest/CreateInvalid
 e3bbe2ace MB-52953: Use only mutation_mem_threshold in VB::hasMemoryForStoredValue
 533286852 MB-53294: Refactor engine Prometheus metrics
 03056b2d2 MB-53294: Rename Cardinality -> MetricGroup
 8937d6e5a MB-52953: Default replication_throttle_threshold=93
 6579346af MB-52956: Update lastReadSeqno at the end of an OSO backfill
 3af167ac7 MB-52953: Move VBucket::mutationMemThreshold to KVBucket
 8c5af9915 MB-52854: Fix and re-enable the DcpConsumerBufferAckTest suite
 100a5b2af MB-52957: Avoid scan when collection high seqno < start
 cd6df9b81 Make wasFirst in ActiveStream snapshot functions const
 7bc7ee427 Sanity check that snap start > previous snap end
 8f324c470 MB-53184: Extend range-scan computed exclusive-end upto the input
 3f6fb6ba2 MB-46738: Remove Vbid arg from the buffer-ack DCP api
 cdc3c2f29 MB-52842: Fix intermittent failure in 'disk>RAM delete paged-out'
 1588cb007 Merge "Merge branch 'neo' into 'master'"
 552d9e2c7 MB-46738: Remove unused dcp_conn_buffer_size_max
 e44ee005e MB-46738: Remove unused dcp_conn_buffer_size
 769d20940 MB-52264: Add desiredMaxSize stat
 6809d7eae MB-46738: Ensure Consumer buffer size always ratio of bucket quota
 b05ebef25 Merge branch 'neo' into 'master'
 ab1ab27f8 Merge "Merge commit 'ea65052e' into 'couchbase/master'"
 a6e70fdae Merge commit 'ea65052e' into 'couchbase/master'
 503ae084b MB-46738: Make DcpFlowControlManager::engine const
 e5766a51e MB-46738: Make dcp_conn_buffer_ratio dynamic
 979159649 MB-53205: Hold VBucket stateLock while calling fetchValidValue
 89602bce3 Humpty-Dumpty: Failover exploration tool
 bb17d9439 MB-53197: [Refactor] create BucketManager::setClusterConfig
 a81e37998 Upgrade go version to 1.19 for tls_test
 256c78709 Merge "MB-52383: Merge branch 'cheshire-cat' into neo" into neo
 09bbfce5c Merge "MB-47851: Merge branch 'cheshire-cat' into neo" into neo
 e99ce1c4a Merge "MB-47267: Merge branch 'cheshire-cat' into neo" into neo
 112e09c36 Merge "MB-51373: Merge branch 'cheshire-cat' into neo" into neo
 281df3be1 Merge "Merge branch 'cheshire-cat' into neo" into neo
 46014c72f MB-52383: Merge branch 'cheshire-cat' into neo
 ecc2f6bb7 Change the logic for Unmetered privilege
 c73eaf5f5 MB-53100: Add streamName arg to MockActiveStream ctor
 ba7850f07 MB-47851: Merge branch 'cheshire-cat' into neo
 5edb02327 MB-47267: Merge branch 'cheshire-cat' into neo
 8db209a68 MB-51373: Merge branch 'cheshire-cat' into neo
 eb865cbb0 Merge branch 'cheshire-cat' into neo
 f656b5152 Merge "Merge branch 'cheshire-cat' into neo" into neo
 453eb9a98 MB-53282: Reset open_time in early return in close_and_rotate_file
 2a83a2a63 MB-52383: Merge branch 'mad-hatter' into cheshire-cat
 9c684fb52 Merge branch 'mad-hatter' into cheshire-cat
 852883091 Merge branch 'mad-hatter' into cheshire-cat
 0173173cb Revert "MB-52813: Don't call Seek for every call of ::scan"
 f1c3ddc67 Merge branch 'cheshire-cat' into neo
 349c2640c Set GOVERSION to 1.18 to remove warning from cmake
 abfb02f80 MB-46738: FCManager API takes DcpConsumer&
 4ab7dbaa3 MB-52264: Wait for memory to reduce before setting new quota
 d5d7b65d0 [serverless] Split Get metering test to individual tests
 fda7ec6b8 Remove old comment in PagingVisitor
 5d9bdbb44 MB-52633: Swap PagingVisitor freq counter histogram to flat array
 f494fa983 MB-51373: Merge branch 'mad-hatter' into cheshire-cat
 6d32e009a MB-52669: Specify GOVERSION without patch revision
 df808528f Merge "Merge branch 'neo'"
 eeb5cbad7 Merge branch 'neo'
 18a4cd691 MB-52793: Merge branch 'mad-hatter' into cheshire-cat
 b4c2fe22b Merge branch 'mad-hatter' into cheshire-cat
 c80c6f58c MB-51373: Inspect and correct Item objects created by KVStore
 ea65052eb MB-53046: [BP] Timeout SeqnoPersistenceRequests when no data is flushed
 5f6d5dc65 MB-47267 / MB-52383: Make backfill during warmup a PauseResume task
 4e51c38a8 MB-47851: Cancel any requests blocked on warmup if warmup stopped.
 2c6e95c8e MB-47267: Make ObjectRegistry getAllocSize atomic
 3d73de526 MB-52902: Populate kvstore rev if no vbstate found
 ad47f53b7 MB-51373: Inspect and correct Item objects created by KVStore
 8855aebe5 MB-52793: Ensure StoredValue::del updates datatype
 35086bc80 Merge remote-tracking branch 'couchbase/alice' into mad-hatter
 0df2087be MB-43055: [BP] Ensure ItemPager available is not left set to false
 6dfd920a8 MB-43453: mcctl: Use passwd from env or stdin
 b7d5bd362 MB-40531: [BP] Prefer paging from replicas if possible
Switching indexing to unstable
indexing is at 19ea81d, changes since last good build: none
Switching plasma to unstable
plasma is at cfa6534, changes since last good build: 
fatal: Invalid revision range 0141641db3ee3de853547c46ed58c647fc7c43a1..HEAD

Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at f2361d1, changes since last good build: none
Pulling in uncommitted change 176468 at refs/changes/68/176468/14
Total 52 (delta 40), reused 47 (delta 40)
[unstable 5aed2c1a] MB-51825: Add NumVBuckets in TsVbuuid struct and fix tsvbuuid pool
 Author: Sai Krishna Teja Kommaraju 
 Date: Tue Jun 21 22:27:17 2022 +0530
 1 file changed, 44 insertions(+), 9 deletions(-)
Pulling in uncommitted change 176523 at refs/changes/23/176523/3
Total 51 (delta 40), reused 48 (delta 40)
[unstable 9b1d2106] MB-51825: Fix memedb_slice_test adding numVBuckets arg
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Jun 22 16:31:49 2022 +0530
 1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178917 at refs/changes/17/178917/8
[unstable b8136e51] MB-51825: Pass numVBuckets to Storage from indexer
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 15:08:14 2022 +0530
 5 files changed, 56 insertions(+), 26 deletions(-)
Pulling in uncommitted change 178948 at refs/changes/48/178948/10
[unstable 7971e62b] MB-51825: Indexer - Fetch numVBuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 20:53:00 2022 +0530
 2 files changed, 57 insertions(+), 21 deletions(-)
Pulling in uncommitted change 178949 at refs/changes/49/178949/10
[unstable 1e0c2545] MB-51825: Scan_Coordinator - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 22:26:59 2022 +0530
 2 files changed, 14 insertions(+), 11 deletions(-)
Pulling in uncommitted change 178950 at refs/changes/50/178950/11
[unstable 577e8f67] MB-51825: Storage_Manager - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Wed Aug 17 23:33:14 2022 +0530
 6 files changed, 102 insertions(+), 17 deletions(-)
Pulling in uncommitted change 178951 at refs/changes/51/178951/11
[unstable 3fe0958b] MB-51825: Mutation_Manager - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 18 00:06:03 2022 +0530
 4 files changed, 37 insertions(+), 22 deletions(-)
Pulling in uncommitted change 178958 at refs/changes/58/178958/11
[unstable 72e32fb5] MB-51825: TimeKeeper - Fetch numVbuckets from cinfo
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Aug 18 04:46:09 2022 +0530
 3 files changed, 59 insertions(+), 44 deletions(-)
Pulling in uncommitted change 179474 at refs/changes/74/179474/1
Total 74 (delta 60), reused 70 (delta 60)
[unstable c074bb7d] MB100 : Add function to satisfy datastore.Context interface
 Author: Sai Krishna Teja Kommaraju 
 Date: Thu Sep 1 22:25:22 2022 +0530
 4 files changed, 16 insertions(+)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable eec4521] MB-52020: Add Shard Locking APIs
 Author: saptarshi.sen 
 Date: Fri Aug 5 11:27:27 2022 -0700
 4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 9739a4c] MB-52020: Add Shard CopyStats
 Author: saptarshi.sen 
 Date: Wed Aug 10 15:03:32 2022 -0700
 9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 59d382d] MB-52020: Add TransferShard & RestoreShard API
 Author: saptarshi.sen 
 Date: Fri Aug 19 00:30:05 2022 -0700
 5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable c7eb7d4] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
 Author: saptarshi.sen 
 Date: Sat Aug 27 22:23:31 2022 -0700
 6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable a15a1b0] MB-52020: Handle zero segmentSize in persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 11:19:16 2022 -0700
 4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable fee3f71] MB-52020: Add checksum to persistent config snapshot
 Author: saptarshi.sen 
 Date: Wed Aug 31 07:50:00 2022 -0700
 4 files changed, 114 insertions(+), 15 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished

Testing

Started testing at 2022/09/02 02:35:34
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000

Simple Test

Sep 02 02:40:35 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 02 02:44:26 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 02 02:45:11 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 02 02:46:41 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 02 02:47:34 b" 'stop_on_failure': 'True'}"
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 02 02:47:34 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_1'}"
Sep 02 02:47:34 b'-->result: '
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 02 02:47:34 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_2'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 02 02:47:34 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_3'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 02 02:47:34 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_4'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 02:47:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 02:57:31 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 02 02:58:13 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 02 03:01:58 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:05:52 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:08:34 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:08:34 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_5'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_6'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_7'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_8'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 02 03:08:34 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 02 03:08:34 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 02 03:08:34 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 02 03:08:34 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 02 03:08:34 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 02 03:08:34 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 02 03:08:34 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 02 03:08:34 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 02 03:08:34 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"

Unit tests

=== RUN   TestMerger
--- PASS: TestMerger (0.01s)
=== RUN   TestInsert
--- PASS: TestInsert (0.00s)
=== RUN   TestInsertPerf
16000000 items took 29.763671549s -> 537568.0877830937 items/s conflicts 23
--- PASS: TestInsertPerf (29.76s)
=== RUN   TestGetPerf
16000000 items took 5.401702236s -> 2.9620292457749606e+06 items/s
--- PASS: TestGetPerf (5.99s)
=== RUN   TestGetRangeSplitItems
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3351,
"memory_used":            37360992,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749144,
"level1": 187776,
"level2": 47222,
"level3": 11903,
"level4": 2974,
"level5": 735,
"level6": 182,
"level7": 48,
"level8": 12,
"level9": 2,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [35911 108875 357987 462344 598654 671258 781160]
No of items in each range [35911 72964 249112 104357 136310 72604 109902 218840]
--- PASS: TestGetRangeSplitItems (0.59s)
=== RUN   TestBuilder
{
"node_count":             50000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3333,
"memory_used":            1866650928,
"node_allocs":            50000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 37500395,
"level1": 9374070,
"level2": 2344403,
"level3": 586130,
"level4": 146617,
"level5": 36338,
"level6": 9033,
"level7": 2299,
"level8": 539,
"level9": 123,
"level10": 39,
"level11": 9,
"level12": 5,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 9.954860057s to build 50000000 items, 5.0226725e+06 items/sec
Took 693.201997ms to iterate 50000000 items
--- PASS: TestBuilder (10.65s)
=== RUN   TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb/skiplist	47.253s
=== RUN   TestInsert
Set IO Concurrency: 716
--- PASS: TestInsert (0.01s)
=== RUN   TestInsertPerf
20000000 items took 49.431309719s -> 404601.862942599 items/s snapshots_created 210 live_snapshots 1
--- PASS: TestInsertPerf (62.00s)
=== RUN   TestInsertDuplicates
--- PASS: TestInsertDuplicates (0.01s)
=== RUN   TestGetPerf
16000000 items took 11.188772578s -> 1.4300049347200163e+06 items/s
--- PASS: TestGetPerf (12.82s)
=== RUN   TestLoadStoreDisk
Inserting 1000000 items took 1.513861451s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       8,
"next_pointers_per_node": 1.3321,
"memory_used":            57313740,
"node_allocs":            999999,
"node_frees":             0,
"level_node_distribution":{
"level0": 750475,
"level1": 187509,
"level2": 46677,
"level3": 11462,
"level4": 2907,
"level5": 715,
"level6": 190,
"level7": 48,
"level8": 15,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
2022-09-02T03:10:51.986+05:30 [Info] MemDB::StoreToDisk: Unexpected amount of data being persisted dir [db.dump] count per shard [[39804 88146 47888 80654 56585 14456 188193 207080 242220 34974 0 0 0 0 0 0]] total count [1000000] snap count [999999] size per shard [[318432 705168 383104 645232 452680 115648 1505544 1656640 1937760 279792 0 0 0 0 0 0]] total size [8000000] memoryInUse [57313740]
2022-09-02T03:10:52.058+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[39804 88146 47888 80654 56585 14456 188193 207080 242220 34974 0 0 0 0 0 0]] total count [1000000] snap count [999999] size per shard [[318432 705168 383104 645232 452680 115648 1505544 1656640 1937760 279792 0 0 0 0 0 0]] total size [8000000] memoryInUse [57313740]
Storing to disk took 330.092005ms
Loading from disk took 291.417456ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3330,
"memory_used":            57328656,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750089,
"level1": 187546,
"level2": 46769,
"level3": 11662,
"level4": 2997,
"level5": 703,
"level6": 184,
"level7": 38,
"level8": 10,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestLoadStoreDisk (3.44s)
=== RUN   TestConcurrentLoadCloseFragmentation
Done Inserting 2000000 items
frag = NaN%
2022-09-02T03:10:57.465+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[2672 151944 85242 108240 50825 171243 119129 99979 175995 314459 55722 36183 14087 102727 70780 440773]] total count [2000000] snap count [2000000] size per shard [[21376 1215552 681936 865920 406600 1369944 953032 799832 1407960 2515672 445776 289464 112696 821816 566240 3526184]] total size [16000000] memoryInUse [114671996]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentation (5.53s)
=== RUN   TestConcurrentLoadCloseFragmentationSmall
Done Inserting 10 items
frag = NaN%
2022-09-02T03:10:59.100+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationSmall (0.09s)
=== RUN   TestConcurrentLoadCloseFragmentationEmpty
Done Inserting 0 items
frag = NaN%
2022-09-02T03:10:59.197+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationEmpty (0.06s)
=== RUN   TestConcurrentCloseSingleNode
A
--- PASS: TestConcurrentCloseSingleNode (0.00s)
=== RUN   TestStoreDiskShutdown
Inserting 1000000 items took 1.591866603s
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       19,
"next_pointers_per_node": 1.3337,
"memory_used":            57339584,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749769,
"level1": 187619,
"level2": 46921,
"level3": 11775,
"level4": 2953,
"level5": 722,
"level6": 182,
"level7": 49,
"level8": 9,
"level9": 1,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
Prepare returns error:  MemDB instance has been shutdown
--- PASS: TestStoreDiskShutdown (2.36s)
=== RUN   TestDelete
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.2000,
"memory_used":            572,
"node_allocs":            10,
"node_frees":             0,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
{
"node_count":             10,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.2000,
"memory_used":            572,
"node_allocs":            20,
"node_frees":             10,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestDelete (1.00s)
=== RUN   TestGCPerf
final_node_count = 16000, average_live_node_count = 68342, wait_time_for_collection = 20ms
--- PASS: TestGCPerf (36.64s)
=== RUN   TestMemoryInUse
ItemsCount: 5000, MemoryInUse: 297328, NodesCount: 5000
ItemsCount: 0, MemoryInUse: 64, NodesCount: 0
--- PASS: TestMemoryInUse (1.01s)
=== RUN   TestFullScan
Inserting 1600000 items took 2.638705882s
{
"node_count":             1600000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       6,
"next_pointers_per_node": 1.3336,
"memory_used":            91740400,
"node_allocs":            1600000,
"node_frees":             0,
"level_node_distribution":{
"level0": 1199521,
"level1": 300382,
"level2": 75072,
"level3": 18867,
"level4": 4639,
"level5": 1137,
"level6": 289,
"level7": 73,
"level8": 18,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Full iteration of 1600000 items took 257.556223ms
--- PASS: TestFullScan (4.06s)
=== RUN   TestVisitor
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3331,
"memory_used":            57329040,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 750011,
"level1": 187841,
"level2": 46531,
"level3": 11594,
"level4": 3049,
"level5": 729,
"level6": 192,
"level7": 41,
"level8": 9,
"level9": 2,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 79.130347ms to iterate 1000000 items, 1.2637377e+07 items/s
shard - 0 count = 25442, range: 0-25441
shard - 1 count = 37610, range: 25442-63051
shard - 2 count = 21545, range: 63052-84596
shard - 3 count = 29031, range: 84597-113627
shard - 4 count = 18696, range: 113628-132323
shard - 5 count = 24128, range: 132324-156451
shard - 6 count = 105793, range: 156452-262244
shard - 7 count = 60804, range: 262245-323048
shard - 8 count = 17942, range: 323049-340990
shard - 9 count = 12013, range: 340991-353003
shard - 10 count = 62502, range: 353004-415505
shard - 11 count = 50301, range: 415506-465806
shard - 12 count = 26097, range: 465807-491903
shard - 13 count = 64785, range: 491904-556688
shard - 14 count = 42807, range: 556689-599495
shard - 15 count = 25019, range: 599496-624514
shard - 16 count = 13207, range: 624515-637721
shard - 17 count = 50415, range: 637722-688136
shard - 18 count = 3055, range: 688137-691191
shard - 19 count = 9842, range: 691192-701033
shard - 20 count = 15039, range: 701034-716072
shard - 21 count = 20247, range: 716073-736319
shard - 22 count = 27071, range: 736320-763390
shard - 23 count = 61815, range: 763391-825205
shard - 24 count = 66250, range: 825206-891455
shard - 25 count = 63032, range: 891456-954487
shard - 26 count = 19382, range: 954488-973869
shard - 27 count = 26130, range: 973870-999999
shard - 28 count = 0, range: 0-0
shard - 29 count = 0, range: 0-0
shard - 30 count = 0, range: 0-0
shard - 31 count = 0, range: 0-0
--- PASS: TestVisitor (1.53s)
=== RUN   TestVisitorError
--- PASS: TestVisitorError (0.13s)
=== RUN   TestLoadDeltaStoreDisk
snap count = 1000000
2022-09-02T03:11:50.171+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 104061 39234 180101 39811 110754 59093 119300 95316 120418 51620 59049]] total count [978757] snap count [1000000] size per shard [[0 0 0 0 0 832488 313872 1440808 318488 886032 472744 954400 762528 963344 412960 472392]] total size [7830056] memoryInUse [127390636]
Storing to disk took 15.393647843s
Loading from disk took 800.052422ms
{
"node_count":             1000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       0,
"next_pointers_per_node": 1.3346,
"memory_used":            57354336,
"node_allocs":            1000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 749046,
"level1": 188067,
"level2": 47299,
"level3": 11690,
"level4": 2920,
"level5": 723,
"level6": 196,
"level7": 42,
"level8": 11,
"level9": 3,
"level10": 2,
"level11": 1,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Restored 21243
RestoredFailed 463660
--- PASS: TestLoadDeltaStoreDisk (18.29s)
=== RUN   TestExecuteConcurrGCWorkers
--- PASS: TestExecuteConcurrGCWorkers (0.65s)
=== RUN   TestCloseWithActiveIterators
--- PASS: TestCloseWithActiveIterators (6.72s)
=== RUN   TestDiskCorruption
Inserting 100000 items took 106.346529ms
{
"node_count":             100000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       10,
"next_pointers_per_node": 1.3338,
"memory_used":            5734012,
"node_allocs":            99998,
"node_frees":             0,
"level_node_distribution":{
"level0": 74940,
"level1": 18901,
"level2": 4548,
"level3": 1204,
"level4": 303,
"level5": 79,
"level6": 16,
"level7": 5,
"level8": 3,
"level9": 1,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
2022-09-02T03:12:11.788+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[9307 13012 820 3760 10901 18931 12803 5356 493 13371 426 3315 6682 823 0 0]] total count [100000] snap count [100000] size per shard [[74456 104096 6560 30080 87208 151448 102424 42848 3944 106968 3408 26520 53456 6584 0 0]] total size [800000] memoryInUse [5734012]
Storing to disk took 83.472155ms
Loading from disk took 357.50594ms
--- PASS: TestDiskCorruption (0.62s)
=== RUN   TestSnapshotStats
--- PASS: TestSnapshotStats (0.00s)
=== RUN   TestInsertDeleteConcurrent
total items in snapshot: 8393413  items deleted: 1606587
{
"node_count":             8393413,
"soft_deletes":           0,
"read_conflicts":         241,
"insert_conflicts":       2,
"next_pointers_per_node": 1.3334,
"memory_used":            481227412,
"node_allocs":            10000000,
"node_frees":             10000002,
"level_node_distribution":{
"level0": 6295269,
"level1": 1573247,
"level2": 393481,
"level3": 98677,
"level4": 24591,
"level5": 6094,
"level6": 1522,
"level7": 390,
"level8": 103,
"level9": 31,
"level10": 6,
"level11": 2,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
allocs: 20000002 frees: 20000002
--- PASS: TestInsertDeleteConcurrent (18.12s)
=== RUN   TestNodeList
--- PASS: TestNodeList (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/memdb	175.391s
=== RUN   TestInteger
--- PASS: TestInteger (0.00s)
=== RUN   TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN   TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN   TestFloat
--- PASS: TestFloat (0.00s)
=== RUN   TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN   TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN   TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN   TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN   TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN   TestReference
--- PASS: TestReference (0.00s)
=== RUN   TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN   TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN   TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN   TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN   TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN   TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN   TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN   TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN   TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618] 
PASS 
TESTING [0] 
PASS 
TESTING [0.0] 
PASS 
TESTING [0.0000] 
PASS 
TESTING [0.0000000] 
PASS 
TESTING [-0] 
PASS 
TESTING [-0.0] 
PASS 
TESTING [-0.0000] 
PASS 
TESTING [-0.0000000] 
PASS 
TESTING [1] 
PASS 
TESTING [20] 
PASS 
TESTING [3456] 
PASS 
TESTING [7645000] 
PASS 
TESTING [9223372036854775807] 
PASS 
TESTING [9223372036854775806] 
PASS 
TESTING [9223372036854775808] 
PASS 
TESTING [92233720368547758071234000] 
PASS 
TESTING [92233720368547758071234987437653] 
PASS 
TESTING [12300000000000000000000000000000056] 
PASS 
TESTING [12300000000000000000000000000000000] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [210690] 
PASS 
TESTING [90000] 
PASS 
TESTING [123000000] 
PASS 
TESTING [3.60e2] 
PASS 
TESTING [36e2] 
PASS 
TESTING [1.9999999999e10] 
PASS 
TESTING [1.99999e10] 
PASS 
TESTING [1.99999e5] 
PASS 
TESTING [0.00000000000012e15] 
PASS 
TESTING [7.64507352e8] 
PASS 
TESTING [9.2233720368547758071234987437653e31] 
PASS 
TESTING [2650e-1] 
PASS 
TESTING [26500e-1] 
PASS 
TESTING [-1] 
PASS 
TESTING [-20] 
PASS 
TESTING [-3456] 
PASS 
TESTING [-7645000] 
PASS 
TESTING [-9223372036854775808] 
PASS 
TESTING [-9223372036854775807] 
PASS 
TESTING [-9223372036854775806] 
PASS 
TESTING [-9223372036854775809] 
PASS 
TESTING [-92233720368547758071234000] 
PASS 
TESTING [-92233720368547758071234987437653] 
PASS 
TESTING [-12300000000000000000000000000000056] 
PASS 
TESTING [-12300000000000000000000000000000000] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056] 
PASS 
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000] 
PASS 
TESTING [-210690] 
PASS 
TESTING [-90000] 
PASS 
TESTING [-123000000] 
PASS 
TESTING [-3.60e2] 
PASS 
TESTING [-36e2] 
PASS 
TESTING [-1.9999999999e10] 
PASS 
TESTING [-1.99999e10] 
PASS 
TESTING [-1.99999e5] 
PASS 
TESTING [-0.00000000000012e15] 
PASS 
TESTING [-2650e-1] 
PASS 
TESTING [-26500e-1] 
PASS 
TESTING [0.03] 
PASS 
TESTING [198.60] 
PASS 
TESTING [2000045.178] 
PASS 
TESTING [1.7976931348623157e+308] 
PASS 
TESTING [0.000000000000000000890] 
PASS 
TESTING [257953786.9864236576] 
PASS 
TESTING [257953786.9864236576e8] 
PASS 
TESTING [36.912e3] 
PASS 
TESTING [2761.67e0] 
PASS 
TESTING [2761.67e00] 
PASS 
TESTING [2761.67e000] 
PASS 
TESTING [7676546.67e-3] 
PASS 
TESTING [-0.03] 
PASS 
TESTING [-198.60] 
PASS 
TESTING [-2000045.178] 
PASS 
TESTING [-1.7976931348623157e+308] 
PASS 
TESTING [-0.000000000000000000890] 
PASS 
TESTING [-257953786.9864236576] 
PASS 
TESTING [-257953786.9864236576e8] 
PASS 
TESTING [-36.912e3] 
PASS 
TESTING [-2761.67e0] 
PASS 
TESTING [-2761.67e00] 
PASS 
TESTING [-2761.67e000] 
PASS 
TESTING [-7676546.67e-3] 
PASS 
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN   TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN   TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN   TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok  	github.com/couchbase/indexing/secondary/collatejson	0.034s
Initializing write barrier = 8000
=== RUN   TestForestDBIterator
2022-09-02T03:12:35.306+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5234 us

2022-09-02T03:12:35.307+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-02T03:12:35.310+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN   TestForestDBIteratorSeek
2022-09-02T03:12:35.311+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-02T03:12:35.314+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN   TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN   TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN   TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN   TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN   TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN   TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 1m29.063007786s -> 112280.06159446055 items/s
Incr build: 10000000 items took 2m18.958197091s -> 71964.08854852419 items/s
Main Index: {
"node_count":             18000000,
"soft_deletes":           0,
"read_conflicts":         0,
"insert_conflicts":       7,
"next_pointers_per_node": 1.3333,
"memory_used":            1695887740,
"node_allocs":            18000000,
"node_frees":             0,
"level_node_distribution":{
"level0": 13500190,
"level1": 3375549,
"level2": 843074,
"level3": 210794,
"level4": 52734,
"level5": 13348,
"level6": 3245,
"level7": 782,
"level8": 221,
"level9": 42,
"level10": 17,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount":  625000,
"SlowHTCount":  0,
"Conflicts":   0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (228.02s)
=== RUN   TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN   TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN   TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN   TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN   TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN   TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN   TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN   TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN   TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN   TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN   TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN   TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok  	github.com/couchbase/indexing/secondary/indexer	252.055s
=== RUN   TestConnPoolBasicSanity
2022-09-02T03:16:50.502+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-09-02T03:16:50.710+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:16:51.504+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN   TestConnRelease
2022-09-02T03:16:55.507+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-09-02T03:17:35.274+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:17:35.522+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.77s)
=== RUN   TestLongevity
2022-09-02T03:17:39.276+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-09-02T03:18:17.674+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:18:18.292+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.40s)
=== RUN   TestSustainedHighConns
2022-09-02T03:18:21.676+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Allocating 16 Connections
cp.curActConns = 0
Returning 3 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 4 Connections
cp.curActConns = 13
Returning 1 Connections
Allocating 12 Connections
cp.curActConns = 22
Returning 1 Connections
cp.curActConns = 23
Allocating 10 Connections
Returning 1 Connections
cp.curActConns = 32
Returning 3 Connections
Allocating 15 Connections
cp.curActConns = 33
Returning 4 Connections
cp.curActConns = 40
Returning 3 Connections
Allocating 8 Connections
cp.curActConns = 44
Returning 2 Connections
cp.curActConns = 43
Allocating 3 Connections
Returning 4 Connections
cp.curActConns = 42
Allocating 9 Connections
Returning 3 Connections
cp.curActConns = 48
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 55
Returning 4 Connections
cp.curActConns = 63
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 59
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 66
Returning 3 Connections
cp.curActConns = 69
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 71
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 78
Returning 0 Connections
cp.curActConns = 80
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 90
Returning 3 Connections
cp.curActConns = 92
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 93
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 101
Returning 3 Connections
cp.curActConns = 108
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 109
Returning 3 Connections
Allocating 22 Connections
cp.curActConns = 112
Returning 4 Connections
cp.curActConns = 124
Returning 2 Connections
Allocating 13 Connections
cp.curActConns = 124
Returning 1 Connections
cp.curActConns = 134
Returning 0 Connections
Allocating 23 Connections
cp.curActConns = 136
Returning 3 Connections
cp.curActConns = 148
Returning 2 Connections
cp.curActConns = 152
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 160
Returning 4 Connections
cp.curActConns = 161
Allocating 8 Connections
Returning 3 Connections
cp.curActConns = 166
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 167
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 171
Returning 1 Connections
cp.curActConns = 174
Returning 2 Connections
Allocating 15 Connections
cp.curActConns = 182
Returning 3 Connections
cp.curActConns = 184
Returning 2 Connections
Allocating 18 Connections
cp.curActConns = 193
Returning 0 Connections
cp.curActConns = 200
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 197
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 205
Returning 2 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 2 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 215
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 212
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 208
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 213
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 215
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 215
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 0 Connections
cp.curActConns = 217
Allocating 4 Connections
Returning 0 Connections
cp.curActConns = 221
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 218
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 218
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 221
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 220
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 219
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 222
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 221
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 220
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 224
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 224
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-09-02T03:19:16.745+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:19:17.712+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.07s)
=== RUN   TestLowWM
2022-09-02T03:19:20.746+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-09-02T03:20:20.762+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-02T03:21:20.778+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-02T03:21:26.253+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:21:26.780+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN   TestTotalConns
2022-09-02T03:21:30.255+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-09-02T03:21:44.430+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:21:45.261+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.18s)
=== RUN   TestUpdateTickRate
2022-09-02T03:21:48.431+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-09-02T03:22:09.279+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:22:09.443+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok  	github.com/couchbase/indexing/secondary/queryport/client	322.836s
Starting server: attempt 1

Functional tests

2022/09/02 03:24:21 In TestMain()
2022/09/02 03:24:21 otp node fetch error: json: cannot unmarshal string into Go value of type couchbase.Pool
2022/09/02 03:24:21 Initialising services with role: kv,n1ql on node: 127.0.0.1:9000
2022/09/02 03:24:22 Initialising web UI on node: 127.0.0.1:9000
2022/09/02 03:24:22 InitWebCreds, response is: {"newBaseUri":"http://127.0.0.1:9000/"}
2022/09/02 03:24:23 Setting data quota of 1500M and Index quota of 1500M
2022/09/02 03:24:24 Adding node: https://127.0.0.1:19001 with role: kv,index to the cluster
2022/09/02 03:24:32 AddNode: Successfully added node: 127.0.0.1:9001 (role kv,index), response: {"otpNode":"n_1@127.0.0.1"}
2022/09/02 03:24:37 Rebalance progress: 0
2022/09/02 03:24:42 Rebalance progress: 0
2022/09/02 03:24:47 Rebalance progress: 0
2022/09/02 03:24:52 Rebalance progress: 0
2022/09/02 03:24:57 Rebalance progress: 0
2022/09/02 03:25:02 Rebalance progress: 0
2022/09/02 03:25:07 Rebalance progress: 0
2022/09/02 03:25:12 Rebalance progress: 0
2022/09/02 03:25:17 Rebalance progress: 0
2022/09/02 03:25:22 Rebalance progress: 0
2022/09/02 03:25:27 Rebalance progress: 0
2022/09/02 03:25:32 Rebalance progress: 0
2022/09/02 03:25:37 Rebalance progress: 0
2022/09/02 03:25:42 Rebalance failed. See logs for detailed reason. You can try again.
2022/09/02 03:25:42 Error while initialising cluster: AddNodeAndRebalance: Error during rebalance, err: Rebalance failed
panic: Error while initialising cluster: AddNodeAndRebalance: Error during rebalance, err: Rebalance failed


goroutine 1 [running]:
panic({0xe8aee0, 0xc000207190})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/panic.go:941 +0x397 fp=0xc000549ca8 sp=0xc000549be8 pc=0x43b757
log.Panicf({0x1025e70?, 0x20?}, {0xc000549df8?, 0xd?, 0xc00003cf50?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/log/log.go:392 +0x67 fp=0xc000549cf0 sp=0xc000549ca8 pc=0x5bdfc7
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/functionaltests.TestMain(0x4461b1?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:94 +0x456 fp=0xc000549ec8 sp=0xc000549cf0 pc=0xd31a76
main.main()
	_testmain.go:483 +0x1d3 fp=0xc000549f80 sp=0xc000549ec8 pc=0xe0cfd3
runtime.main()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:250 +0x212 fp=0xc000549fe0 sp=0xc000549f80 pc=0x43e2d2
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x46ec21

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060fb0 sp=0xc000060f90 pc=0x43e696
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.forcegchelper()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:301 +0xad fp=0xc000060fe0 sp=0xc000060fb0 pc=0x43e52d
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46ec21
created by runtime.init.6
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:289 +0x25

goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c790 sp=0xc00005c770 pc=0x43e696
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005c7c8 sp=0xc00005c790 pc=0x4297f7
runtime.gcenable.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005c7e0 sp=0xc00005c7c8 pc=0x41f3a6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46ec21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x6b

goroutine 19 [GC scavenge wait]:
runtime.gopark(0x3a9aa9faa4a5?, 0x10000?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cf20 sp=0xc00005cf00 pc=0x43e696
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005cfc8 sp=0xc00005cf20 pc=0x427605
runtime.gcenable.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005cfe0 sp=0xc00005cfc8 pc=0x41f346
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46ec21
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0xaa

goroutine 3 [finalizer wait]:
runtime.gopark(0x0?, 0x109fd68?, 0x60?, 0x0?, 0x2000000020?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060630 sp=0xc000060610 pc=0x43e696
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.runfinq()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000607e0 sp=0xc000060630 pc=0x41e453
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46ec21
created by runtime.createfing
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:157 +0x45

goroutine 34 [select]:
runtime.gopark(0xc000061798?, 0x2?, 0x47?, 0xe7?, 0xc00006178c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061618 sp=0xc0000615f8 pc=0x43e696
runtime.selectgo(0xc000061798, 0xc000061788, 0x0?, 0x0, 0xb?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061758 sp=0xc000061618 pc=0x44e112
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc0002040f0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:396 +0x67 fp=0xc0000617c8 sp=0xc000061758 pc=0x785e07
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x26 fp=0xc0000617e0 sp=0xc0000617c8 pc=0x786a86
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46ec21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x37a

goroutine 35 [select]:
runtime.gopark(0xc000244798?, 0x2?, 0x0?, 0x0?, 0xc00024478c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000244608 sp=0xc0002445e8 pc=0x43e696
runtime.selectgo(0xc000244798, 0xc000244788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000244748 sp=0xc000244608 pc=0x44e112
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc000204108)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:316 +0x85 fp=0xc0002447c8 sp=0xc000244748 pc=0x785825
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x26 fp=0xc0002447e0 sp=0xc0002447c8 pc=0x786a26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002447e8 sp=0xc0002447e0 pc=0x46ec21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x3ca

goroutine 36 [IO wait]:
runtime.gopark(0xc00022c340?, 0xc00004e000?, 0x70?, 0x38?, 0x484542?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000543800 sp=0xc0005437e0 pc=0x43e696
runtime.netpollblock(0xc000145000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000543838 sp=0xc000543800 pc=0x437137
internal/poll.runtime_pollWait(0x7fc6c0444fc0, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000543858 sp=0xc000543838 pc=0x469209
internal/poll.(*pollDesc).wait(0xc000124100?, 0xc000145000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000543880 sp=0xc000543858 pc=0x4a2132
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000124100, {0xc000145000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000543900 sp=0xc000543880 pc=0x4a349a
net.(*netFD).Read(0xc000124100, {0xc000145000?, 0xc00051e990?, 0xc0005439d0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000543948 sp=0xc000543900 pc=0x669209
net.(*conn).Read(0xc00011a010, {0xc000145000?, 0x11?, 0xc000543a68?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000543990 sp=0xc000543948 pc=0x679485
bufio.(*Reader).Read(0xc0001141e0, {0xc000274001, 0x5ff, 0x453934?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc0005439c8 sp=0xc000543990 pc=0x5206f4
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x191?, {0xc000274001?, 0x8?, 0x20?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc0005439f8 sp=0xc0005439c8 pc=0x7d7045
encoding/json.(*Decoder).refill(0xc000225400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000543a48 sp=0xc0005439f8 pc=0x565bbf
encoding/json.(*Decoder).readValue(0xc000225400)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000543a98 sp=0xc000543a48 pc=0x5657bb
encoding/json.(*Decoder).Decode(0xc000225400, {0xebbc00, 0xc000200380})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000543ac8 sp=0xc000543a98 pc=0x565418
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc000200360, 0xc00020e4e0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc000543b08 sp=0xc000543ac8 pc=0x7d62a5
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc000216690?, 0x4cd388?)
	:1 +0x2a fp=0xc000543b28 sp=0xc000543b08 pc=0x7d8dca
net/rpc.(*Server).readRequestHeader(0xc000216690, {0x11b5fe8, 0xc000206610})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:587 +0x66 fp=0xc000543bf8 sp=0xc000543b28 pc=0x7d59c6
net/rpc.(*Server).readRequest(0x0?, {0x11b5fe8, 0xc000206610})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:547 +0x3b fp=0xc000543cd0 sp=0xc000543bf8 pc=0x7d551b
net/rpc.(*Server).ServeCodec(0xc000216690, {0x11b5fe8?, 0xc000206610})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:462 +0x87 fp=0xc000543dc8 sp=0xc000543cd0 pc=0x7d4c47
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000244f60?, 0xc00012cfa0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc000543f38 sp=0xc000543dc8 pc=0x7d7799
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0x11ac700?, 0xc000122030?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc000543f70 sp=0xc000543f38 pc=0x7d7e98
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc00023e000)
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc000543fc0 sp=0xc000543f70 pc=0x7e1c1d
github.com/couchbase/cbauth.startDefault.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc000543fe0 sp=0xc000543fc0 pc=0x7e1f05
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x46ec21
created by github.com/couchbase/cbauth.startDefault
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9

goroutine 11 [select]:
runtime.gopark(0xc000128f90?, 0x2?, 0xd8?, 0x8d?, 0xc000128f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000128d90 sp=0xc000128d70 pc=0x43e696
runtime.selectgo(0xc000128f90, 0xc000128f20, 0xc0004a27c0?, 0x0, 0xc00049fbc0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000128ed0 sp=0xc000128d90 pc=0x44e112
net/http.(*persistConn).writeLoop(0xc00054a000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000128fc8 sp=0xc000128ed0 pc=0x76d1b5
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000128fe0 sp=0xc000128fc8 pc=0x769c46
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000128fe8 sp=0xc000128fe0 pc=0x46ec21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 10 [IO wait]:
runtime.gopark(0xc0004a81a0?, 0xc000052a00?, 0x68?, 0x5b?, 0x484542?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000075af8 sp=0xc000075ad8 pc=0x43e696
runtime.netpollblock(0xc0004ac000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000075b30 sp=0xc000075af8 pc=0x437137
internal/poll.runtime_pollWait(0x7fc6c0444ed0, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000075b50 sp=0xc000075b30 pc=0x469209
internal/poll.(*pollDesc).wait(0xc000540180?, 0xc0004ac000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000075b78 sp=0xc000075b50 pc=0x4a2132
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000540180, {0xc0004ac000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000075bf8 sp=0xc000075b78 pc=0x4a349a
net.(*netFD).Read(0xc000540180, {0xc0004ac000?, 0x40bae9?, 0x4?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000075c40 sp=0xc000075bf8 pc=0x669209
net.(*conn).Read(0xc000010888, {0xc0004ac000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000075c88 sp=0xc000075c40 pc=0x679485
net/http.(*persistConn).Read(0xc00054a000, {0xc0004ac000?, 0xc0000466c0?, 0xc000075d30?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000075ce8 sp=0xc000075c88 pc=0x76a6ae
bufio.(*Reader).fill(0xc0001e9440)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000075d20 sp=0xc000075ce8 pc=0x520123
bufio.(*Reader).Peek(0xc0001e9440, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:144 +0x5d fp=0xc000075d40 sp=0xc000075d20 pc=0x52027d
net/http.(*persistConn).readLoop(0xc00054a000)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2093 +0x1ac fp=0xc000075fc8 sp=0xc000075d40 pc=0x76b4cc
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000075fe0 sp=0xc000075fc8 pc=0x769ca6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x46ec21
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 66 [GC worker (idle)]:
runtime.gopark(0xe5da60?, 0xc00003d5e7?, 0x16?, 0x0?, 0x11b5fe8?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000240758 sp=0xc000240738 pc=0x43e696
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0002407e0 sp=0xc000240758 pc=0x421485
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002407e8 sp=0xc0002407e0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 56 [GC worker (idle)]:
runtime.gopark(0x3a9aa9df1640?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000240f58 sp=0xc000240f38 pc=0x43e696
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000240fe0 sp=0xc000240f58 pc=0x421485
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000240fe8 sp=0xc000240fe0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 55 [GC worker (idle)]:
runtime.gopark(0xe5da60?, 0xc00050c317?, 0x16?, 0x0?, 0x11b5fe8?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062758 sp=0xc000062738 pc=0x43e696
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0000627e0 sp=0xc000062758 pc=0x421485
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 20 [GC worker (idle)]:
runtime.gopark(0xc000244fa0?, 0xc000244fb0?, 0x1d?, 0x1c?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000244f58 sp=0xc000244f38 pc=0x43e696
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000244fe0 sp=0xc000244f58 pc=0x421485
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000244fe8 sp=0xc000244fe0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
signal: aborted (core dumped)
FAIL	github.com/couchbase/indexing/secondary/tests/functionaltests	81.003s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
2022/09/02 03:25:44 In TestMain()
2022/09/02 03:25:44 ChangeIndexerSettings: Host  Port 0 Nodes []
2022/09/02 03:25:44 Changing config key indexer.api.enableTestServer to value true
2022/09/02 03:25:44 Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused


goroutine 1 [running]:
panic({0xcd25e0, 0xc00050c9a0})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/panic.go:941 +0x397 fp=0xc0001d3d40 sp=0xc0001d3c80 pc=0x43a6d7
log.Panicf({0xe3f566?, 0x1e?}, {0xc0001d3e38?, 0x1c?, 0xcc7ce0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/log/log.go:392 +0x67 fp=0xc0001d3d88 sp=0xc0001d3d40 pc=0x5bb407
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/largedatatests.TestMain(0x445131?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/largedatatests/common_test.go:52 +0x468 fp=0xc0001d3ec8 sp=0xc0001d3d88 pc=0xc5e768
main.main()
	_testmain.go:59 +0x1d3 fp=0xc0001d3f80 sp=0xc0001d3ec8 pc=0xc64833
runtime.main()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:250 +0x212 fp=0xc0001d3fe0 sp=0xc0001d3f80 pc=0x43d252
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001d3fe8 sp=0xc0001d3fe0 pc=0x46dba1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060fb0 sp=0xc000060f90 pc=0x43d616
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.forcegchelper()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:301 +0xad fp=0xc000060fe0 sp=0xc000060fb0 pc=0x43d4ad
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46dba1
created by runtime.init.6
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:289 +0x25

goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c790 sp=0xc00005c770 pc=0x43d616
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005c7c8 sp=0xc00005c790 pc=0x428777
runtime.gcenable.func1()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005c7e0 sp=0xc00005c7c8 pc=0x41e326
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46dba1
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x6b

goroutine 19 [GC scavenge wait]:
runtime.gopark(0x3aab86428e9e?, 0x10000?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cf20 sp=0xc00005cf00 pc=0x43d616
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005cfc8 sp=0xc00005cf20 pc=0x426585
runtime.gcenable.func2()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005cfe0 sp=0xc00005cfc8 pc=0x41e2c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46dba1
created by runtime.gcenable
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0xaa

goroutine 20 [finalizer wait]:
runtime.gopark(0x0?, 0xea3598?, 0xc0?, 0x40?, 0x2000000020?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060630 sp=0xc000060610 pc=0x43d616
runtime.goparkunlock(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.runfinq()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000607e0 sp=0xc000060630 pc=0x41d3d3
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46dba1
created by runtime.createfing
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:157 +0x45

goroutine 3 [select]:
runtime.gopark(0xc000061798?, 0x2?, 0x0?, 0x0?, 0xc00006178c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061618 sp=0xc0000615f8 pc=0x43d616
runtime.selectgo(0xc000061798, 0xc000061788, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061758 sp=0xc000061618 pc=0x44d092
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc00000e0f0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:396 +0x67 fp=0xc0000617c8 sp=0xc000061758 pc=0x779647
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x26 fp=0xc0000617e0 sp=0xc0000617c8 pc=0x77a2c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46dba1
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x37a

goroutine 4 [select]:
runtime.gopark(0xc000061f98?, 0x2?, 0x0?, 0x0?, 0xc000061f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061e08 sp=0xc000061de8 pc=0x43d616
runtime.selectgo(0xc000061f98, 0xc000061f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061f48 sp=0xc000061e08 pc=0x44d092
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc00000e108)
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:316 +0x85 fp=0xc000061fc8 sp=0xc000061f48 pc=0x779065
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x26 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x77a266
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x46dba1
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
	/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x3ca

goroutine 5 [IO wait]:
runtime.gopark(0xc0000036c0?, 0xc00004e000?, 0x70?, 0x98?, 0x483482?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00015d800 sp=0xc00015d7e0 pc=0x43d616
runtime.netpollblock(0xc0001a9000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00015d838 sp=0xc00015d800 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6b0d8, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00015d858 sp=0xc00015d838 pc=0x468189
internal/poll.(*pollDesc).wait(0xc00012c180?, 0xc0001a9000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00015d880 sp=0xc00015d858 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00012c180, {0xc0001a9000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00015d900 sp=0xc00015d880 pc=0x4a211a
net.(*netFD).Read(0xc00012c180, {0xc0001a9000?, 0xc0004743f0?, 0xc?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc00015d948 sp=0xc00015d900 pc=0x665589
net.(*conn).Read(0xc00012a4f8, {0xc0001a9000?, 0x11?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc00015d990 sp=0xc00015d948 pc=0x674aa5
bufio.(*Reader).Read(0xc0001144e0, {0xc000038601, 0x5ff, 0x4528b4?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc00015d9c8 sp=0xc00015d990 pc=0x51dd14
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x203000?, {0xc000038601?, 0x203000?, 0xc0004327e0?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc00015d9f8 sp=0xc00015d9c8 pc=0x7b9da5
encoding/json.(*Decoder).refill(0xc000480280)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc00015da48 sp=0xc00015d9f8 pc=0x562fff
encoding/json.(*Decoder).readValue(0xc000480280)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc00015da98 sp=0xc00015da48 pc=0x562bfb
encoding/json.(*Decoder).Decode(0xc000480280, {0xcfd5c0, 0xc0003bef80})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc00015dac8 sp=0xc00015da98 pc=0x562858
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc0003bef60, 0xc0004327e0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc00015db08 sp=0xc00015dac8 pc=0x7b9005
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc000099900?, 0x4cc008?)
	:1 +0x2a fp=0xc00015db28 sp=0xc00015db08 pc=0x7bbb2a
net/rpc.(*Server).readRequestHeader(0xc000099900, {0xf8d6e8, 0xc000474370})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:587 +0x66 fp=0xc00015dbf8 sp=0xc00015db28 pc=0x7b8726
net/rpc.(*Server).readRequest(0x0?, {0xf8d6e8, 0xc000474370})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:547 +0x3b fp=0xc00015dcd0 sp=0xc00015dbf8 pc=0x7b827b
net/rpc.(*Server).ServeCodec(0xc000099900, {0xf8d6e8?, 0xc000474370})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:462 +0x87 fp=0xc00015ddc8 sp=0xc00015dcd0 pc=0x7b79a7
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000062760?, 0xc00006efa0)
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc00015df38 sp=0xc00015ddc8 pc=0x7ba4f9
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0xf84480?, 0xc000136618?})
	/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc00015df70 sp=0xc00015df38 pc=0x7babf8
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc0000b2000)
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc00015dfc0 sp=0xc00015df70 pc=0x7c44fd
github.com/couchbase/cbauth.startDefault.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc00015dfe0 sp=0xc00015dfc0 pc=0x7c47e5
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00015dfe8 sp=0xc00015dfe0 pc=0x46dba1
created by github.com/couchbase/cbauth.startDefault
	/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9

goroutine 6 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062e78 sp=0xc000062e58 pc=0x43d616
runtime.chanrecv(0xc0003beea0, 0xc000062f90, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc000062f08 sp=0xc000062e78 pc=0x40b5cc
runtime.chanrecv2(0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:445 +0x18 fp=0xc000062f30 sp=0xc000062f08 pc=0x40b038
github.com/couchbase/goutils/systemeventlog.(*SystemEventLoggerImpl).logEvents(0xc0000d9880)
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:186 +0xb7 fp=0xc000062fc8 sp=0xc000062f30 pc=0xb001b7
github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger.func1()
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x26 fp=0xc000062fe0 sp=0xc000062fc8 pc=0xaffac6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x46dba1
created by github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger
	/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x1d6

goroutine 8 [select]:
runtime.gopark(0xc0001d9ba0?, 0x3?, 0x0?, 0x30?, 0xc0001d9b1a?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001d9978 sp=0xc0001d9958 pc=0x43d616
runtime.selectgo(0xc0001d9ba0, 0xc0001d9b14, 0x3?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001d9ab8 sp=0xc0001d9978 pc=0x44d092
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc000076840?, {0xe504e2, 0x1b}, 0xc0001d9e68, 0xc000042600)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:301 +0x429 fp=0xc0001d9e40 sp=0xc0001d9ab8 pc=0x9b8289
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:259
github.com/couchbase/cbauth/metakv.RunObserveChildren({0xe504e2?, 0x0?}, 0x0?, 0x0?)
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x58 fp=0xc0001d9e88 sp=0xc0001d9e40 pc=0x9b8838
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0?, {0x0?, 0x0?})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1579 +0xc7 fp=0xc0001d9f00 sp=0xc0001d9e88 pc=0xb05147
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc0001d9fa0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc0001d9f38 sp=0xc0001d9f00 pc=0xace643
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1584 +0xdf fp=0xc0001d9fe0 sp=0xc0001d9f38 pc=0xb04fff
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001d9fe8 sp=0xc0001d9fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1572 +0xaf

goroutine 15 [IO wait]:
runtime.gopark(0xc0002381a0?, 0xc00004e000?, 0x68?, 0x2b?, 0x483482?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000202af8 sp=0xc000202ad8 pc=0x43d616
runtime.netpollblock(0xc0004f6000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000202b30 sp=0xc000202af8 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6afe8, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000202b50 sp=0xc000202b30 pc=0x468189
internal/poll.(*pollDesc).wait(0xc0004da280?, 0xc0004f6000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000202b78 sp=0xc000202b50 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004da280, {0xc0004f6000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000202bf8 sp=0xc000202b78 pc=0x4a211a
net.(*netFD).Read(0xc0004da280, {0xc0004f6000?, 0x8?, 0xc000560c88?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000202c40 sp=0xc000202bf8 pc=0x665589
net.(*conn).Read(0xc00020e098, {0xc0004f6000?, 0xc000010038?, 0xc00003d2c0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000202c88 sp=0xc000202c40 pc=0x674aa5
net/http.(*persistConn).Read(0xc00022e7e0, {0xc0004f6000?, 0x40a1fd?, 0x60?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000202ce8 sp=0xc000202c88 pc=0x76588e
bufio.(*Reader).fill(0xc000201080)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000202d20 sp=0xc000202ce8 pc=0x51d743
bufio.(*Reader).Peek(0xc000201080, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:144 +0x5d fp=0xc000202d40 sp=0xc000202d20 pc=0x51d89d
net/http.(*persistConn).readLoop(0xc00022e7e0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2093 +0x1ac fp=0xc000202fc8 sp=0xc000202d40 pc=0x7666ac
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000202fe0 sp=0xc000202fc8 pc=0x764e86
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000202fe8 sp=0xc000202fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 22 [GC worker (idle)]:
runtime.gopark(0xc0000627a0?, 0xc0000627b0?, 0xfd?, 0x44?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062758 sp=0xc000062738 pc=0x43d616
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0000627e0 sp=0xc000062758 pc=0x420405
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x3aab863b9083?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005df58 sp=0xc00005df38 pc=0x43d616
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005dfe0 sp=0xc00005df58 pc=0x420405
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x3aab863b5e3d?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005e758 sp=0xc00005e738 pc=0x43d616
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005e7e0 sp=0xc00005e758 pc=0x420405
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x3aab863d292a?, 0x0?, 0x0?, 0x0?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005ef58 sp=0xc00005ef38 pc=0x43d616
runtime.gcBgMarkWorker()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005efe0 sp=0xc00005ef58 pc=0x420405
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25

goroutine 53 [runnable]:
runtime.gopark(0xc0001dc9c0?, 0x6?, 0x80?, 0xc6?, 0xc0001dc7d4?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001dc638 sp=0xc0001dc618 pc=0x43d616
runtime.selectgo(0xc0001dc9c0, 0xc0001dc7c8, 0xe45617?, 0x0, 0xc0001dc7b0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001dc778 sp=0xc0001dc638 pc=0x44d092
net/http.(*persistConn).roundTrip(0xc00043aea0, 0xc000214300)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2620 +0x974 fp=0xc0001dca30 sp=0xc0001dc778 pc=0x769254
net/http.(*Transport).roundTrip(0x170e3e0, 0xc000226800)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:594 +0x7c9 fp=0xc0001dcc68 sp=0xc0001dca30 pc=0x75cce9
net/http.(*Transport).RoundTrip(0xc000226800?, 0xf83c00?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc0001dcc88 sp=0xc0001dcc68 pc=0x744f19
net/http.send(0xc000226700, {0xf83c00, 0x170e3e0}, {0xdceb60?, 0x48e901?, 0x1787d40?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:252 +0x5d8 fp=0xc0001dce68 sp=0xc0001dcc88 pc=0x706818
net/http.(*Client).send(0xc00020a9c0, 0xc000226700, {0xc0002284c0?, 0xc0001dcf70?, 0x1787d40?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:176 +0x9b fp=0xc0001dcee0 sp=0xc0001dce68 pc=0x7060bb
net/http.(*Client).do(0xc00020a9c0, 0xc000226700)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:725 +0x8f5 fp=0xc0001dd0e0 sp=0xc0001dcee0 pc=0x7084f5
net/http.(*Client).Do(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc00021e9c0?, 0x3b?}, 0xc0001dd470, {0x0, 0x0}, 0x0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0001dd1e0 sp=0xc0001dd0e0 pc=0x8626c9
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc0001de000, {0xc000248510, 0x26}, {0xc0002101c0?, 0xc0001dd690?}, {0xcb0740, 0xc000218378}, 0xcb6bc0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc0001dd428 sp=0xc0001dd1e0 pc=0x884093
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.(*Pool).GetServerGroups(0xc000144878)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:907 +0xbd fp=0xc0001dd498 sp=0xc0001dd428 pc=0x88837d
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchServerGroups(0xc000144780)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:694 +0x4e fp=0xc0001dd778 sp=0xc0001dd498 pc=0xab06ce
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x7f7a7bb1c5b8?, {0x0?, 0x0?})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:381 +0x62f fp=0xc0001dde78 sp=0xc0001dd778 pc=0xaad12f
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc0001ddec8)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc0001ddeb0 sp=0xc0001dde78 pc=0xace643
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0xc00003d2c0?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:402 +0x6b fp=0xc0001ddf00 sp=0xc0001ddeb0 pc=0xaacacb
github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache({0xc00003d2c0?, 0x0?}, {0xe3fad7, 0x7}, {0xe4b8b8, 0x16})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:138 +0x15b fp=0xc0001ddf40 sp=0xc0001ddf00 pc=0xaab9db
github.com/couchbase/indexing/secondary/common.MonitorInternalVersion(0x0?, {0xe3e4ae, 0x5}, {0xc00003d2c0, 0xe})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:791 +0x5b fp=0xc0001ddfa8 sp=0xc0001ddf40 pc=0xacc1fb
github.com/couchbase/indexing/secondary/queryport/client.watchClusterVer.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:2046 +0x3b fp=0xc0001ddfe0 sp=0xc0001ddfa8 pc=0xc257fb
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001ddfe8 sp=0xc0001ddfe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.watchClusterVer
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:2046 +0xd5

goroutine 54 [select]:
runtime.gopark(0xc000565f98?, 0x2?, 0x0?, 0x30?, 0xc000565f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000565e18 sp=0xc000565df8 pc=0x43d616
runtime.selectgo(0xc000565f98, 0xc000565f88, 0x0?, 0x0, 0x0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000565f58 sp=0xc000565e18 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*schedTokenMonitor).updater(0xc0001161e0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2389 +0x92 fp=0xc000565fc8 sp=0xc000565f58 pc=0xc37f92
github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x26 fp=0xc000565fe0 sp=0xc000565fc8 pc=0xc368c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x2e5

goroutine 57 [select]:
runtime.gopark(0xc0001daea8?, 0x6?, 0x0?, 0x30?, 0xc0001dacbc?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001dab20 sp=0xc0001dab00 pc=0x43d616
runtime.selectgo(0xc0001daea8, 0xc0001dacb0, 0xe45617?, 0x0, 0xc0001daef0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001dac60 sp=0xc0001dab20 pc=0x44d092
net/http.(*persistConn).roundTrip(0xc00022e7e0, 0xc0002143c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2620 +0x974 fp=0xc0001daf18 sp=0xc0001dac60 pc=0x769254
net/http.(*Transport).roundTrip(0x170e3e0, 0xc000226a00)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:594 +0x7c9 fp=0xc0001db150 sp=0xc0001daf18 pc=0x75cce9
net/http.(*Transport).RoundTrip(0xc000226a00?, 0xf83c00?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc0001db170 sp=0xc0001db150 pc=0x744f19
net/http.send(0xc000226900, {0xf83c00, 0x170e3e0}, {0xdceb60?, 0x48e901?, 0x1787d40?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:252 +0x5d8 fp=0xc0001db350 sp=0xc0001db170 pc=0x706818
net/http.(*Client).send(0xc00020ab70, 0xc000226900, {0x7f7a7ba5e868?, 0x150?, 0x1787d40?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:176 +0x9b fp=0xc0001db3c8 sp=0xc0001db350 pc=0x7060bb
net/http.(*Client).do(0xc00020ab70, 0xc000226900)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:725 +0x8f5 fp=0xc0001db5c8 sp=0xc0001db3c8 pc=0x7084f5
net/http.(*Client).Do(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc000216540?, 0x1b?}, 0xc0001db960, {0x0, 0x0}, 0x0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0001db6c8 sp=0xc0001db5c8 pc=0x8626c9
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc000222a20, {0xe3ebec, 0x6}, {0xd43bc0?, 0xc000534001?}, {0xcb0680, 0xc0004da198}, 0xc00021ec00?)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc0001db910 sp=0xc0001db6c8 pc=0x884093
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth({0xc00021ec00, 0x3f}, {0xf83820?, 0xc000210260})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:594 +0x125 fp=0xc0001db988 sp=0xc0001db910 pc=0x886045
github.com/couchbase/indexing/secondary/dcp.Connect({0xc00021ec00, 0x3f})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:600 +0xce fp=0xc0001dbb20 sp=0xc0001db988 pc=0x8861ee
github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier({0xc00021ec00, 0x3f}, {0xe3fad7, 0x7}, {0xe42d8c, 0xb})
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:239 +0x1a8 fp=0xc0001dbda0 sp=0xc0001dbb20 pc=0xad0f48
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).watchClusterChanges(0xc0000e02c0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2071 +0xff fp=0xc0001dbfc8 sp=0xc0001dbda0 pc=0xc35c9f
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x26 fp=0xc0001dbfe0 sp=0xc0001dbfc8 pc=0xc29426
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001dbfe8 sp=0xc0001dbfe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x5c5

goroutine 30 [IO wait]:
runtime.gopark(0xc000436000?, 0xc000050500?, 0x10?, 0xa?, 0x483482?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0000709a0 sp=0xc000070980 pc=0x43d616
runtime.netpollblock(0xc0004dc000?, 0x1000?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0000709d8 sp=0xc0000709a0 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6ad18, 0x72)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0000709f8 sp=0xc0000709d8 pc=0x468189
internal/poll.(*pollDesc).wait(0xc0004da080?, 0xc0004dc000?, 0x0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000070a20 sp=0xc0000709f8 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004da080, {0xc0004dc000, 0x1000, 0x1000})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000070aa0 sp=0xc000070a20 pc=0x4a211a
net.(*netFD).Read(0xc0004da080, {0xc0004dc000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000070ae8 sp=0xc000070aa0 pc=0x665589
net.(*conn).Read(0xc00020e048, {0xc0004dc000?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000070b30 sp=0xc000070ae8 pc=0x674aa5
net/http.(*persistConn).Read(0xc00022e360, {0xc0004dc000?, 0x1000?, 0x1000?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000070b90 sp=0xc000070b30 pc=0x76588e
bufio.(*Reader).fill(0xc000200ae0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000070bc8 sp=0xc000070b90 pc=0x51d743
bufio.(*Reader).ReadSlice(0xc000200ae0, 0x38?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:371 +0x2f fp=0xc000070c18 sp=0xc000070bc8 pc=0x51e32f
net/http/internal.readChunkLine(0xc0001e8ce8?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc000070c68 sp=0xc000070c18 pc=0x7036c5
net/http/internal.(*chunkedReader).beginChunk(0xc000124990)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc000070c98 sp=0xc000070c68 pc=0x703148
net/http/internal.(*chunkedReader).Read(0xc000124990, {0xc000538a00?, 0x0?, 0x0?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc000070d18 sp=0xc000070c98 pc=0x70340e
net/http.(*body).readLocked(0xc000094640, {0xc000538a00?, 0x73fb7d?, 0xc0001e8e80?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:844 +0x3c fp=0xc000070d68 sp=0xc000070d18 pc=0x75a3fc
net/http.(*body).Read(0x1000000000000?, {0xc000538a00?, 0x0?, 0x7f7a7bb1c5b8?})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:836 +0x125 fp=0xc000070de0 sp=0xc000070d68 pc=0x75a2c5
net/http.(*bodyEOFSignal).Read(0xc0000949c0, {0xc000538a00, 0x200, 0x200})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2774 +0x142 fp=0xc000070e60 sp=0xc000070de0 pc=0x769fc2
encoding/json.(*Decoder).refill(0xc0001423c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000070eb0 sp=0xc000070e60 pc=0x562fff
encoding/json.(*Decoder).readValue(0xc0001423c0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000070f00 sp=0xc000070eb0 pc=0x562bfb
encoding/json.(*Decoder).Decode(0xc0001423c0, {0xcaee80, 0xc000098320})
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000070f30 sp=0xc000070f00 pc=0x562858
github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1()
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x10b fp=0xc000070fe0 sp=0xc000070f30 pc=0x9b872b
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x46dba1
created by github.com/couchbase/cbauth/metakv.doRunObserveChildren
	/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:280 +0x2eb

goroutine 16 [select]:
runtime.gopark(0xc000207f90?, 0x2?, 0xd8?, 0x7d?, 0xc000207f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000207d90 sp=0xc000207d70 pc=0x43d616
runtime.selectgo(0xc000207f90, 0xc000207f20, 0xc000214440?, 0x0, 0xc00020ad20?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000207ed0 sp=0xc000207d90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00022e7e0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000207fc8 sp=0xc000207ed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000207fe0 sp=0xc000207fc8 pc=0x764e26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000207fe8 sp=0xc000207fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 28 [select]:
runtime.gopark(0xc000206f68?, 0x4?, 0x3?, 0x0?, 0xc000206db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000206c00 sp=0xc000206be0 pc=0x43d616
runtime.selectgo(0xc000206f68, 0xc000206da8, 0xc000094a80?, 0x0, 0xc000565501?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000206d40 sp=0xc000206c00 pc=0x44d092
net/http.(*persistConn).readLoop(0xc00043aea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc000206fc8 sp=0xc000206d40 pc=0x7672a5
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000206fe0 sp=0xc000206fc8 pc=0x764e86
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000206fe8 sp=0xc000206fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 29 [select]:
runtime.gopark(0xc00006ff90?, 0x2?, 0xd8?, 0xfd?, 0xc00006ff24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006fd90 sp=0xc00006fd70 pc=0x43d616
runtime.selectgo(0xc00006ff90, 0xc00006ff20, 0xc000094480?, 0x0, 0xc00020aa50?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00006fed0 sp=0xc00006fd90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00043aea0)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00006ffc8 sp=0xc00006fed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x764e26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 11 [select]:
runtime.gopark(0xc000571f68?, 0x4?, 0x3?, 0x0?, 0xc000571db0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000571c00 sp=0xc000571be0 pc=0x43d616
runtime.selectgo(0xc000571f68, 0xc000571da8, 0xc000094640?, 0x0, 0xc000560d01?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000571d40 sp=0xc000571c00 pc=0x44d092
net/http.(*persistConn).readLoop(0xc00022e360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc000571fc8 sp=0xc000571d40 pc=0x7672a5
net/http.(*Transport).dialConn.func5()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x764e86
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e

goroutine 12 [select]:
runtime.gopark(0xc000209f90?, 0x2?, 0xd8?, 0x9d?, 0xc000209f24?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000209d90 sp=0xc000209d70 pc=0x43d616
runtime.selectgo(0xc000209f90, 0xc000209f20, 0xc000214180?, 0x0, 0xc00020a660?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000209ed0 sp=0xc000209d90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00022e360)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000209fc8 sp=0xc000209ed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000209fe0 sp=0xc000209fc8 pc=0x764e26
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000209fe8 sp=0xc000209fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791

goroutine 58 [chan receive]:
runtime.gopark(0xc00005d6d8?, 0x4431bb?, 0x20?, 0xd7?, 0x459dc5?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005d6c8 sp=0xc00005d6a8 pc=0x43d616
runtime.chanrecv(0xc0003bf560, 0x0, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc00005d758 sp=0xc00005d6c8 pc=0x40b5cc
runtime.chanrecv1(0xdf8475800?, 0x0?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:440 +0x18 fp=0xc00005d780 sp=0xc00005d758 pc=0x40aff8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).logstats(0xc0000e02c0)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:1410 +0x79 fp=0xc00005d7c8 sp=0xc00005d780 pc=0xc30a39
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x26 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0xc293c6
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x605

goroutine 59 [select]:
runtime.gopark(0xc000564f90?, 0x2?, 0x40?, 0xc?, 0xc000564f8c?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000564e18 sp=0xc000564df8 pc=0x43d616
runtime.selectgo(0xc000564f90, 0xc000564f88, 0x0?, 0x0, 0xf8d598?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000564f58 sp=0xc000564e18 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).listenMetaChange(0xc000032c00, 0xc0000b0380)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1830 +0x70 fp=0xc000564fc0 sp=0xc000564f58 pc=0xc24930
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func1()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x2a fp=0xc000564fe0 sp=0xc000564fc0 pc=0xc2488a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000564fe8 sp=0xc000564fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x28a

goroutine 60 [select]:
runtime.gopark(0xc00005f750?, 0x2?, 0x0?, 0x30?, 0xc00005f714?)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005f590 sp=0xc00005f570 pc=0x43d616
runtime.selectgo(0xc00005f750, 0xc00005f710, 0xe6f1ce?, 0x0, 0xcd1be0?, 0x1)
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00005f6d0 sp=0xc00005f590 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).logstats(0xc000032c00, 0xc0000b0380)
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1849 +0x238 fp=0xc00005f7c0 sp=0xc00005f6d0 pc=0xc24bb8
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func2()
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2a fp=0xc00005f7e0 sp=0xc00005f7c0 pc=0xc2482a
runtime.goexit()
	/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
	/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2ed
signal: aborted (core dumped)
FAIL	github.com/couchbase/indexing/secondary/tests/largedatatests	0.141s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused

Integration tests

echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[  0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
 'conf_file': 'conf/simple_gsi_n1ql.conf',
 'gsi_type': 'memory_optimized',
 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
 'makefile': 'True',
 'num_nodes': 4,
 'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:27 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-02 03:26:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-02 03:26:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused 
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1948-rel-EE-enterprise
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 suite_setUp==============
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster 
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-02 03:26:55 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5484658688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:55 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:56 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 5313880064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:56 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 5189267456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '18', 'memoryTotal': 15466930176, 'memoryFree': 5093285888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-02 03:26:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-02 03:27:03 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-02 03:28:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.277509736964467, 'mem_free': 13542510592, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 102.04294872283936
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:15 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-02 03:28:15 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 145.374686ms
2022-09-02 03:28:47 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'faed4653-404b-4a47-8aa9-b8380b9a0aca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '145.374686ms', 'executionTime': '145.147882ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 671.018664ms
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.488701ms
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.996234ms
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.603157ms
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.537300159591324, 'mem_free': 13455237120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-02 03:28:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:28:55 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 148.913s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result: 
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1948-rel-EE-enterprise
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.428749156005401, 'mem_free': 13450723328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:29:01 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '151', 'memoryTotal': 15466930176, 'memoryFree': 13424906240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1 
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-02 03:29:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '149', 'memoryTotal': 15466930176, 'memoryFree': 13435834368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '150', 'memoryTotal': 15466930176, 'memoryFree': 13435478016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute]  {'uptime': '149', 'memoryTotal': 15466930176, 'memoryFree': 13426843648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:05 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-02 03:29:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [basetestcase.setUp] ==============  basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.496182174896917, 'mem_free': 13424103424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 74.18479132652283
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1  is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs.  ...
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:16 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-02 03:30:16 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs.  ...
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.537338ms
2022-09-02 03:30:48 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'd4c3aeb8-8f2a-4a22-9902-74b7973d8d47', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.537338ms', 'executionTime': '62.476347ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default 
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 645.309013ms
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.361302ms
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-02 03:30:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeecbb9fb710ec44faa9ea4295a8527c075job_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeecbb9fb710ec44faa9ea4295a8527c075job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.75446ms
2022-09-02 03:30:50 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeecbb9fb710ec44faa9ea4295a8527c075job_title) USING GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeecbb9fb710ec44faa9ea4295a8527c075job_title) USING GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeecbb9fb710ec44faa9ea4295a8527c075job_title%29+USING+GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 24.016115ms
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.288459ms
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.261519ms
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.444519ms
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'eb8b0326-c7ef-40ba-b002-28ae7f8092df', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeecbb9fb710ec44faa9ea4295a8527c075job_title', 'index_id': 'e8ce974648532c66', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.444519ms', 'executionTime': '2.37641ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is   doc["job_title"] == "Sales" 
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause=  doc["job_title"] == "Sales" 
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 116.603404ms
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Actual Result
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results]  Analyzing Expected Result
2022-09-02 03:30:57 | INFO | MainProcess | Cluster_Thread | [task.execute]  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:57 | INFO | MainProcess | Cluster_Thread | [task.check]  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.732816ms
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeecbb9fb710ec44faa9ea4295a8527c075job_title ON default USING GSI
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeecbb9fb710ec44faa9ea4295a8527c075job_title+ON+default+USING+GSI
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 37.547272ms
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.384594ms
2022-09-02 03:30:58 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list.  List output: {'requestID': 'ee8a75b4-8d06-40c1-8b2c-109f25bf9ff0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.384594ms', 'executionTime': '5.327267ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.246185ms
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.795454ms
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.707996668458969, 'mem_free': 13347934208, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:04 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:31:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:31:05 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ==============  basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1
ok

----------------------------------------------------------------------
Ran 1 test in 129.948s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2'}
[2022-09-02 03:31:05,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:05,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:06,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:06,292] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:31:06,373] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:31:06,374] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:31:06,375] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:31:07,026] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:31:07,055] - [task:164] INFO -  {'uptime': '276', 'memoryTotal': 15466930176, 'memoryFree': 13315837952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:07,083] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:31:07,083] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:31:07,084] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:31:07,117] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:31:07,155] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:31:07,156] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:31:07,183] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:31:07,184] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:31:07,185] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:31:07,185] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:31:07,234] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:07,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:07,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:07,492] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:07,493] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:07,559] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:07,560] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:07,591] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:07,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:07,714] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:31:07,740] - [task:164] INFO -  {'uptime': '275', 'memoryTotal': 15466930176, 'memoryFree': 13346504704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:07,765] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:07,792] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:31:07,792] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:31:07,843] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:07,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:07,848] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:08,100] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:08,101] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:08,169] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:08,170] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:08,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:08,203] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:08,320] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:31:08,347] - [task:164] INFO -  {'uptime': '275', 'memoryTotal': 15466930176, 'memoryFree': 13346648064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:08,372] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:08,400] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:31:08,400] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:31:08,452] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:08,455] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:08,455] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:08,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:08,709] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:08,775] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:08,776] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:08,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:08,806] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:08,923] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:31:08,950] - [task:164] INFO -  {'uptime': '269', 'memoryTotal': 15466930176, 'memoryFree': 13346852864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:08,977] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:09,005] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:31:09,005] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:31:09,057] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:09,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:09,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:09,314] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:09,315] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:09,381] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:09,382] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:09,411] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:09,412] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:09,501] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:31:09,884] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:31:14,888] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:31:14,973] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:31:14,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:14,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:15,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:15,239] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:15,306] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:15,307] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:15,307] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:31:15,613] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:31:15,669] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:31:15,670] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:32:05,579] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:05,941] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:06,239] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:32:06,241] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:32:06,300] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,300] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:06,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:06,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:06,825] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:06,830] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:07,076] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:07,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:07,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:07,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:11,128] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:32:11,128] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.215339608572898, 'mem_free': 13328629760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:32:11,128] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:32:11,129] - [basetestcase:467] INFO - Time to execute basesetup : 65.13436388969421
[2022-09-02 03:32:11,180] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,180] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:32:11,233] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,233] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:32:11,287] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,287] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:32:11,347] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,347] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:32:11,403] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:11,464] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:32:11,465] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:32:11,465] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:32:16,472] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:32:16,475] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:16,475] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:16,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:17,719] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:32:17,883] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:20,543] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:32:20,558] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:32:21,224] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:21,224] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:32:21,224] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:32:51,255] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:32:51,284] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:32:51,309] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:32:51,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.935505ms
[2022-09-02 03:32:51,380] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b2ed11eb-34f3-4452-8bc6-a2d37ebb2339', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.935505ms', 'executionTime': '68.876283ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:32:51,380] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:32:51,407] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:32:51,432] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:32:52,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 680.564482ms
[2022-09-02 03:32:52,115] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:32:52,181] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:32:52,212] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:32:52,221] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.43582ms
[2022-09-02 03:32:52,455] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:52,491] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:32:52,504] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:32:52,506] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:32:52,517] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:32:52,576] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:53,400] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2d203bfa69194995af9d55b941b297ccjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:32:53,427] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2d203bfa69194995af9d55b941b297ccjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:32:53,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.754369ms
[2022-09-02 03:32:53,459] - [base_gsi:282] INFO - BUILD INDEX on default(employee2d203bfa69194995af9d55b941b297ccjob_title) USING GSI
[2022-09-02 03:32:54,490] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee2d203bfa69194995af9d55b941b297ccjob_title) USING GSI
[2022-09-02 03:32:54,515] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee2d203bfa69194995af9d55b941b297ccjob_title%29+USING+GSI
[2022-09-02 03:32:54,536] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.133108ms
[2022-09-02 03:32:55,565] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:32:55,590] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:32:55,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.184288ms
[2022-09-02 03:32:56,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:32:56,654] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:32:56,663] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.368873ms
[2022-09-02 03:32:57,178] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-02 03:32:57,344] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:58,196] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:32:58,666] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:32:58,699] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:32:58,726] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:32:58,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.117003ms
[2022-09-02 03:32:58,731] - [task:3245] INFO - {'requestID': '169620b7-bebe-47e4-acb5-8ce8ac038719', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2d203bfa69194995af9d55b941b297ccjob_title', 'index_id': 'e689fc1e0c806489', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.117003ms', 'executionTime': '3.039588ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:32:58,731] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:32:58,731] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:32:58,732] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:32:58,732] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:32:58,733] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-02 03:32:58,733] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:32:58,733] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:32:58,733] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:32:58,733] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:32:59,733] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:32:59,763] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:32:59,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:32:59,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 113.002601ms
[2022-09-02 03:32:59,912] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:32:59,912] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:33:00,723] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:33:00,723] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:33:01,755] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:33:01,782] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:33:01,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.101188ms
[2022-09-02 03:33:01,819] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2d203bfa69194995af9d55b941b297ccjob_title ON default USING GSI
[2022-09-02 03:33:01,848] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2d203bfa69194995af9d55b941b297ccjob_title+ON+default+USING+GSI
[2022-09-02 03:33:01,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.009202ms
[2022-09-02 03:33:01,928] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:33:01,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:33:01,964] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.43114ms
[2022-09-02 03:33:01,964] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '11076b53-8242-464f-b40d-bb6e0441a38b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.43114ms', 'executionTime': '5.361373ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:33:02,077] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:02,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:02,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:02,462] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:02,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:33:02,519] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:33:02,585] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:33:02,643] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:02,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:02,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:03,024] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:03,082] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:33:03,082] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:33:03,143] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:33:03,202] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:03,202] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:33:03,202] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:33:03,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:33:03,255] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:33:03,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.28082ms
[2022-09-02 03:33:03,287] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:33:03,312] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:33:03,369] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.143543ms
[2022-09-02 03:33:03,430] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:33:03,430] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.72944656850717, 'mem_free': 13206835200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:33:03,431] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:33:03,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:03,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:03,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:03,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:03,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:04,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:04,134] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:04,134] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:04,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:04,690] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:04,691] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:05,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:09,330] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:33:09,472] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:33:10,346] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:33:10,375] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:33:10,375] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:33:10,431] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,485] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,539] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,540] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:33:10,632] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:33:10,633] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:33:10,659] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,790] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:33:10,790] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,816] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:33:10,841] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:33:10,841] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,868] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:33:10,894] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:33:10,894] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,920] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:33:10,945] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:33:10,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,972] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:33:10,972] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:33:10,973] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:33:10,973] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2
ok

----------------------------------------------------------------------
Ran 1 test in 125.030s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3'}
[2022-09-02 03:33:11,069] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:11,069] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:11,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:11,411] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:11,488] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:33:11,488] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:33:11,489] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:33:12,098] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:33:12,125] - [task:164] INFO -  {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13275095040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:12,151] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:33:12,151] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:33:12,152] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:33:12,183] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:33:12,222] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:33:12,223] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:33:12,251] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:33:12,300] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:12,303] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:12,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:12,608] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:12,609] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:12,684] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:12,686] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:12,715] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:12,716] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:12,842] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:33:12,868] - [task:164] INFO -  {'uptime': '400', 'memoryTotal': 15466930176, 'memoryFree': 13218377728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:12,895] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:12,923] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:33:12,923] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:33:12,974] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:12,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:12,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:13,281] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:13,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:13,357] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:13,358] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:13,391] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:13,391] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:13,504] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:33:13,530] - [task:164] INFO -  {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13217734656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:13,556] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:13,583] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:33:13,583] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:33:13,634] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:13,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:13,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:13,962] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:13,963] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:14,036] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:14,037] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:14,067] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:14,068] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:14,186] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:33:14,212] - [task:164] INFO -  {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13219360768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:14,238] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:14,265] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:33:14,265] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:33:14,316] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:14,320] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:14,320] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:14,629] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:14,630] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:14,703] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:14,704] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:14,735] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:14,736] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:14,823] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:33:15,218] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:33:20,219] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:33:20,303] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:33:20,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:20,308] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:20,640] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:20,641] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:20,716] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:20,717] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:20,718] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:33:20,934] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:33:20,993] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:33:20,994] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:34:05,622] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:05,995] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:06,302] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:34:06,304] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:34:06,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:06,378] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:06,704] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:06,708] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:06,709] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,003] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:07,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:07,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,347] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:07,357] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:07,358] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:11,956] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:34:11,956] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.93267907264082, 'mem_free': 13270089728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:34:11,956] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:34:11,956] - [basetestcase:467] INFO - Time to execute basesetup : 60.889695167541504
[2022-09-02 03:34:12,007] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,008] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:34:12,060] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,061] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:34:12,114] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,114] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:34:12,167] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,167] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:34:12,222] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:12,282] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:34:12,283] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:34:12,283] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:34:17,294] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:34:17,299] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:17,299] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:17,649] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:18,661] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:34:18,832] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:21,395] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:34:21,414] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:34:22,258] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:22,259] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:34:22,259] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:34:52,288] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:34:52,317] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:34:52,342] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:34:52,409] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.558727ms
[2022-09-02 03:34:52,409] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'af56e504-e83d-4978-bf7e-cbe460c0aa34', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.558727ms', 'executionTime': '64.442284ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:34:52,409] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:34:52,435] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:34:52,461] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:34:53,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 671.889114ms
[2022-09-02 03:34:53,135] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:34:53,192] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:34:53,234] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:34:53,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.805583ms
[2022-09-02 03:34:53,443] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:53,481] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:34:53,495] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:34:53,496] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:34:53,514] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:34:53,585] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:54,407] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee02be61b6900f45fd81049c85068a0a9ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:34:54,432] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee02be61b6900f45fd81049c85068a0a9ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:34:54,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.431676ms
[2022-09-02 03:34:54,466] - [base_gsi:282] INFO - BUILD INDEX on default(employee02be61b6900f45fd81049c85068a0a9ajob_title) USING GSI
[2022-09-02 03:34:55,501] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee02be61b6900f45fd81049c85068a0a9ajob_title) USING GSI
[2022-09-02 03:34:55,526] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee02be61b6900f45fd81049c85068a0a9ajob_title%29+USING+GSI
[2022-09-02 03:34:55,548] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.614095ms
[2022-09-02 03:34:56,580] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:34:56,608] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:34:56,618] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.068854ms
[2022-09-02 03:34:57,648] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:34:57,673] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:34:57,680] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.566114ms
[2022-09-02 03:34:58,155] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:34:58,323] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:59,023] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:34:59,683] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:34:59,716] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:34:59,744] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:34:59,748] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.280499ms
[2022-09-02 03:34:59,748] - [task:3245] INFO - {'requestID': '84496e26-bef4-4720-9d43-a5b9cd9e93cc', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee02be61b6900f45fd81049c85068a0a9ajob_title', 'index_id': 'ab2090ba08dbba3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.280499ms', 'executionTime': '2.20169ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-02 03:34:59,749] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:34:59,749] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:34:59,749] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:34:59,750] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:34:59,750] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-02 03:34:59,750] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:34:59,751] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:34:59,751] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:34:59,751] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:35:00,751] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:35:00,781] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:35:00,808] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:35:00,907] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 89.700283ms
[2022-09-02 03:35:00,907] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:35:00,908] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:35:01,714] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:35:01,714] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:35:02,748] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:35:02,777] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:35:02,785] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.468333ms
[2022-09-02 03:35:02,814] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee02be61b6900f45fd81049c85068a0a9ajob_title ON default USING GSI
[2022-09-02 03:35:02,840] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee02be61b6900f45fd81049c85068a0a9ajob_title+ON+default+USING+GSI
[2022-09-02 03:35:02,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.02186ms
[2022-09-02 03:35:02,931] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:35:02,959] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:35:02,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.603696ms
[2022-09-02 03:35:02,967] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '26f2d73b-fead-46fa-8d65-ccbdacb07507', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.603696ms', 'executionTime': '5.531127ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:35:03,082] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:03,086] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:03,086] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:03,471] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:03,527] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:35:03,527] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:35:03,596] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:35:03,651] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:03,654] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:03,654] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:04,046] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:04,110] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:35:04,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:35:04,183] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:35:04,241] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:04,241] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:35:04,242] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:35:04,269] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:35:04,296] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:35:04,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.598169ms
[2022-09-02 03:35:04,329] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:35:04,355] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:35:04,405] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.501231ms
[2022-09-02 03:35:04,474] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:35:04,474] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.47086309230359, 'mem_free': 13138907136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:35:04,474] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:35:04,479] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:04,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:04,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:04,879] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:04,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:05,473] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:05,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:05,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:06,197] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:06,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:06,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:06,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:11,166] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:35:11,318] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:35:12,270] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:35:12,298] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:35:12,298] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:35:12,353] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,408] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,476] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,477] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:35:12,559] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:35:12,560] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:35:12,587] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,719] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:35:12,719] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,749] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:35:12,777] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:35:12,777] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,804] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:35:12,830] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:35:12,831] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,858] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:35:12,885] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:35:12,885] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,914] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:35:12,914] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:35:12,914] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:35:12,915] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3
ok

----------------------------------------------------------------------
Ran 1 test in 121.900s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4'}
[2022-09-02 03:35:13,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:13,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:13,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:13,424] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:13,504] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:35:13,505] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:35:13,505] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:35:14,047] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:35:14,075] - [task:164] INFO -  {'uptime': '523', 'memoryTotal': 15466930176, 'memoryFree': 13237760000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:14,103] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:35:14,103] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:35:14,104] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:35:14,145] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:35:14,183] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:35:14,184] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:35:14,215] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:35:14,268] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:14,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:14,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:14,675] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:14,676] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:14,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:14,763] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:14,793] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:14,794] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:14,918] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:35:14,947] - [task:164] INFO -  {'uptime': '521', 'memoryTotal': 15466930176, 'memoryFree': 13132648448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:14,974] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:15,003] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:35:15,003] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:35:15,056] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:15,059] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:15,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:15,446] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:15,447] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:15,533] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:15,534] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:15,568] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:15,568] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:15,685] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:35:15,711] - [task:164] INFO -  {'uptime': '521', 'memoryTotal': 15466930176, 'memoryFree': 13231337472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:15,737] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:15,765] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:35:15,766] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:35:15,819] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:15,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:15,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:16,219] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:16,220] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:16,314] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:16,315] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:16,347] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:16,348] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:16,477] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:35:16,507] - [task:164] INFO -  {'uptime': '520', 'memoryTotal': 15466930176, 'memoryFree': 13246390272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:16,535] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:16,564] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:35:16,564] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:35:16,618] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:16,621] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:16,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:17,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:17,038] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:17,131] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:17,132] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:17,165] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:17,166] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:17,256] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:35:17,674] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:35:22,676] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:35:22,766] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:35:22,772] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:22,772] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:23,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:23,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:23,277] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:23,278] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:23,279] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:35:24,372] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:35:24,429] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:35:24,430] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:36:05,406] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:05,722] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:06,239] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:36:06,242] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:36:06,303] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:06,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:06,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:06,694] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:06,694] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:07,048] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:07,055] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:07,055] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:07,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:07,598] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:07,598] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:08,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:12,702] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:36:12,702] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.72395422371751, 'mem_free': 13239820288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:36:12,702] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:36:12,703] - [basetestcase:467] INFO - Time to execute basesetup : 59.6883327960968
[2022-09-02 03:36:12,758] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,758] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:36:12,810] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,810] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:36:12,863] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,864] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:36:12,918] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,918] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:36:12,969] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:13,035] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:36:13,035] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:36:13,035] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:36:18,043] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:36:18,046] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:18,046] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:18,414] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:19,438] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:36:19,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:22,127] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:36:22,150] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:36:22,833] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:22,833] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:36:22,833] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:36:52,862] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:36:52,896] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:36:52,925] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:36:52,997] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.047189ms
[2022-09-02 03:36:52,998] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '0c341982-0299-4f21-9e3e-5ba4b32756a4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.047189ms', 'executionTime': '70.980201ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:36:52,998] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:36:53,027] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:36:53,054] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:36:53,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.542504ms
[2022-09-02 03:36:53,738] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:36:53,816] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:36:53,851] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:36:53,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.031534ms
[2022-09-02 03:36:54,080] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:54,118] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:36:54,136] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:36:54,139] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:36:54,174] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:36:54,236] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:55,037] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee815de908b3294035881b89a82c2cb77fjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:36:55,065] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee815de908b3294035881b89a82c2cb77fjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:36:55,106] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.8135ms
[2022-09-02 03:36:55,107] - [base_gsi:282] INFO - BUILD INDEX on default(employee815de908b3294035881b89a82c2cb77fjob_title) USING GSI
[2022-09-02 03:36:56,138] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee815de908b3294035881b89a82c2cb77fjob_title) USING GSI
[2022-09-02 03:36:56,163] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee815de908b3294035881b89a82c2cb77fjob_title%29+USING+GSI
[2022-09-02 03:36:56,191] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.952335ms
[2022-09-02 03:36:57,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:36:57,248] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:36:57,257] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.04057ms
[2022-09-02 03:36:58,288] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:36:58,316] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:36:58,325] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.737056ms
[2022-09-02 03:36:58,792] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:36:58,967] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:59,773] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:36:59,972] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-02 03:36:59,972] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:37:00,153] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-02 03:37:00,154] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-02 03:37:00,154] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2086350291, 0, b'')
[2022-09-02 03:37:00,327] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:00,356] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:37:00,384] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:37:00,389] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.002272ms
[2022-09-02 03:37:00,389] - [task:3245] INFO - {'requestID': '1b29e3a3-1887-4aa1-b689-3652399e56fd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee815de908b3294035881b89a82c2cb77fjob_title', 'index_id': '7c3fa2286002c776', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.002272ms', 'executionTime': '2.941184ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:37:00,389] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:00,389] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:00,390] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:37:00,390] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:37:00,390] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-02 03:37:00,391] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:37:00,391] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:37:00,391] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:37:00,391] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:37:01,391] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:01,421] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:37:01,449] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:37:01,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 112.537957ms
[2022-09-02 03:37:01,570] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:37:01,571] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:37:02,400] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:02,400] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:37:03,432] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:37:03,459] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:37:03,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.490259ms
[2022-09-02 03:37:03,495] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee815de908b3294035881b89a82c2cb77fjob_title ON default USING GSI
[2022-09-02 03:37:03,524] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee815de908b3294035881b89a82c2cb77fjob_title+ON+default+USING+GSI
[2022-09-02 03:37:03,563] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.321182ms
[2022-09-02 03:37:03,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:37:03,628] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:37:03,635] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.210414ms
[2022-09-02 03:37:03,635] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '04d1b611-644d-41e8-a8b8-0a786b850b8d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.210414ms', 'executionTime': '5.148652ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:37:03,741] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:03,745] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:03,745] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:04,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:04,302] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:37:04,303] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:37:04,379] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:37:04,437] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:04,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:04,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:04,949] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:05,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:37:05,017] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:37:05,096] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:37:05,156] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:05,157] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:37:05,157] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:37:05,185] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:37:05,215] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:37:05,222] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.43078ms
[2022-09-02 03:37:05,251] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:37:05,279] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:37:05,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.490743ms
[2022-09-02 03:37:05,409] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:37:05,410] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.38479004483664, 'mem_free': 13098631168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:37:05,410] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:37:05,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:05,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:05,904] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:05,909] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:05,909] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:06,396] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:06,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:06,402] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:06,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:06,871] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:06,871] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:07,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:13,396] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:37:13,540] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:37:14,305] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:37:14,334] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:37:14,334] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:37:14,389] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,442] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,496] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,497] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:37:14,576] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:37:14,578] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:37:14,605] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,754] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:37:14,754] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,780] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:37:14,806] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:37:14,806] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,833] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:37:14,858] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:37:14,859] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,886] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:37:14,911] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:37:14,911] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,938] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:37:14,938] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:37:14,939] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:37:14,939] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4
ok

----------------------------------------------------------------------
Ran 1 test in 121.977s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5'}
[2022-09-02 03:37:15,038] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:15,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:15,457] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:15,488] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:15,570] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:37:15,570] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:37:15,571] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:37:16,068] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:37:16,099] - [task:164] INFO -  {'uptime': '645', 'memoryTotal': 15466930176, 'memoryFree': 13202739200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:16,124] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:37:16,124] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:37:16,125] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:37:16,157] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:37:16,195] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:37:16,195] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:37:16,223] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:37:16,224] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:37:16,224] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:37:16,225] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:37:16,273] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:16,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:16,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:16,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:16,709] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:16,807] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:16,808] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:16,840] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:16,841] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:16,973] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:37:17,002] - [task:164] INFO -  {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13124141056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:17,028] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:17,055] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:37:17,056] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:37:17,109] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:17,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:17,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:17,600] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:17,601] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:17,703] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:17,704] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:17,737] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:17,737] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:17,860] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:37:17,890] - [task:164] INFO -  {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13202726912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:17,917] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:17,946] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:37:17,947] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:37:18,002] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:18,007] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:18,007] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:18,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:18,481] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:18,575] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:18,576] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:18,608] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:18,609] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:18,724] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:37:18,751] - [task:164] INFO -  {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13202874368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:18,777] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:18,804] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:37:18,804] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:37:18,856] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:18,860] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:18,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:19,332] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:19,333] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:19,431] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:19,432] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:19,461] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:19,461] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:19,554] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:37:20,003] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:37:25,006] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:37:25,095] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:37:25,100] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:25,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:25,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:25,550] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:25,641] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:25,642] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:25,642] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:37:26,707] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:37:26,762] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:37:26,763] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:38:05,692] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:06,014] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:06,297] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:38:06,300] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:38:06,362] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:06,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:06,785] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:06,790] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:06,791] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:07,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:07,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:07,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:07,940] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:07,947] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:07,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:08,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:13,815] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:38:13,816] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.94294453438017, 'mem_free': 13204148224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:38:13,816] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:38:13,816] - [basetestcase:467] INFO - Time to execute basesetup : 58.7807400226593
[2022-09-02 03:38:13,867] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,867] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:38:13,920] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,921] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:38:13,977] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,977] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:38:14,036] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:14,036] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:38:14,086] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:14,144] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:38:14,144] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:38:14,144] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:38:19,153] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:38:19,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:19,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:19,585] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:20,670] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:38:20,839] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:23,256] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:38:23,272] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-02 03:38:24,348] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:24,348] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:38:24,348] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:38:54,377] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:38:54,406] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:38:54,432] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:38:54,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.149085ms
[2022-09-02 03:38:54,497] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '7e8e72ad-145b-4767-8337-02d1bba709a7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.149085ms', 'executionTime': '62.083293ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:38:54,497] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:38:54,523] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:38:54,550] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:38:55,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 711.546631ms
[2022-09-02 03:38:55,264] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:38:55,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:38:55,365] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:38:55,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.168973ms
[2022-09-02 03:38:55,582] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:55,618] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:38:55,633] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:38:55,634] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:38:55,649] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:38:55,721] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:56,545] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee7c05dfae775b4e799ad9a6cf48fc842ajob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:38:56,571] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:38:56,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.416821ms
[2022-09-02 03:38:56,605] - [base_gsi:282] INFO - BUILD INDEX on default(employee7c05dfae775b4e799ad9a6cf48fc842ajob_title) USING GSI
[2022-09-02 03:38:57,635] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee7c05dfae775b4e799ad9a6cf48fc842ajob_title) USING GSI
[2022-09-02 03:38:57,660] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%29+USING+GSI
[2022-09-02 03:38:57,685] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.860941ms
[2022-09-02 03:38:58,717] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:38:58,744] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:38:58,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.341981ms
[2022-09-02 03:38:59,787] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:38:59,816] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:38:59,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.062601ms
[2022-09-02 03:39:00,310] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:39:00,476] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:39:04,078] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:39:04,828] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:04,863] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:39:04,889] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:39:04,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.030947ms
[2022-09-02 03:39:04,895] - [task:3245] INFO - {'requestID': '784be84b-e467-40dd-b7e8-bd4cfea9e8df', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title', 'index_id': '33279191c77201c6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.030947ms', 'executionTime': '2.962671ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:39:04,895] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:04,896] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:04,896] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:39:04,897] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:39:04,897] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-02 03:39:04,897] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:39:04,898] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:39:04,898] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:39:04,898] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:39:05,897] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:05,930] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:39:05,956] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:39:06,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 145.034406ms
[2022-09-02 03:39:06,115] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:39:06,116] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:39:07,729] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:07,729] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:39:08,760] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:39:08,785] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:39:08,792] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.679802ms
[2022-09-02 03:39:08,819] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee7c05dfae775b4e799ad9a6cf48fc842ajob_title ON default USING GSI
[2022-09-02 03:39:08,844] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee7c05dfae775b4e799ad9a6cf48fc842ajob_title+ON+default+USING+GSI
[2022-09-02 03:39:08,886] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.153314ms
[2022-09-02 03:39:08,922] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:39:08,947] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:39:08,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.345381ms
[2022-09-02 03:39:08,955] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1e9259d3-86aa-4fa1-945c-cc51fc2161bb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.345381ms', 'executionTime': '5.271523ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:39:09,059] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:09,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:09,062] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:09,575] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:09,631] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:39:09,631] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:39:09,722] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:39:09,778] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:09,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:09,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:10,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:10,377] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:39:10,377] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:39:10,470] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:39:10,526] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:10,526] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:39:10,526] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:39:10,551] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:39:10,577] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:39:10,584] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.794432ms
[2022-09-02 03:39:10,610] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:39:10,636] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:39:10,684] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.93437ms
[2022-09-02 03:39:10,759] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:39:10,760] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.14824495541436, 'mem_free': 13057966080, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:39:10,760] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:39:10,764] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:10,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:11,328] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:11,333] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:11,333] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:11,878] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:11,883] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:11,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:12,694] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:12,702] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:12,702] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:13,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:19,758] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:39:19,901] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:39:20,297] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:39:20,325] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:39:20,325] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:39:20,379] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,433] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,488] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,489] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:39:20,570] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:39:20,571] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:39:20,598] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,732] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:39:20,733] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,759] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:39:20,785] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:39:20,785] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,812] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:39:20,837] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:39:20,838] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,864] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:39:20,890] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:39:20,890] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,917] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:39:20,917] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:39:20,917] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:39:20,918] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5
ok

----------------------------------------------------------------------
Ran 1 test in 125.935s

OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi

Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6'}
[2022-09-02 03:39:21,034] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:21,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:21,525] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:21,557] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:21,641] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:39:21,641] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:39:21,642] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:39:22,058] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:39:22,086] - [task:164] INFO -  {'uptime': '771', 'memoryTotal': 15466930176, 'memoryFree': 13143281664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:22,112] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:39:22,112] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:39:22,113] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:39:22,146] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:39:22,180] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:39:22,180] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:39:22,210] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:39:22,211] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:39:22,212] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:39:22,212] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:39:22,260] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:22,263] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:22,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:22,768] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:22,769] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:22,873] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:22,874] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:22,908] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:22,909] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:23,038] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:39:23,064] - [task:164] INFO -  {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 13145874432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:23,090] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:23,116] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:39:23,117] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:39:23,168] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:23,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:23,171] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:23,660] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:23,661] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:23,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:23,764] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:23,794] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:23,794] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:23,910] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:39:23,936] - [task:164] INFO -  {'uptime': '767', 'memoryTotal': 15466930176, 'memoryFree': 13146128384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:23,966] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:23,993] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:39:23,994] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:39:24,045] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:24,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:24,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:24,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:24,551] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:24,651] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:24,652] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:24,682] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:24,682] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:24,800] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:39:24,828] - [task:164] INFO -  {'uptime': '767', 'memoryTotal': 15466930176, 'memoryFree': 13144543232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:24,854] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:24,882] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:39:24,883] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:39:24,934] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:24,936] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:24,937] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:25,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:25,456] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:25,557] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:25,558] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:25,587] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:25,587] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:25,681] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:39:26,087] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:39:31,090] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:39:31,173] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:39:31,178] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:31,179] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:31,719] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:31,720] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:31,831] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:31,832] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:31,833] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:39:32,814] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:39:32,872] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:39:32,873] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:40:05,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:05,737] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:06,255] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:40:06,259] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:40:06,317] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:06,317] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:06,891] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:06,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:06,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:07,588] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:07,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:07,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:08,495] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:08,503] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:08,503] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:09,422] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:14,999] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:40:15,000] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.67130366951604, 'mem_free': 13144801280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:40:15,000] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:40:15,000] - [basetestcase:467] INFO - Time to execute basesetup : 53.97261834144592
[2022-09-02 03:40:15,055] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,055] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:40:15,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,109] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:40:15,161] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,161] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:40:15,215] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,215] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:40:15,266] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:15,326] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:40:15,327] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:40:15,327] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:40:20,333] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:40:20,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:20,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:20,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:21,881] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:40:22,150] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:24,524] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:40:24,544] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-02 03:40:25,935] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:25,936] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:40:25,936] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:40:55,951] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:40:55,980] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:40:56,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:40:56,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.970307ms
[2022-09-02 03:40:56,070] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'd4587a4b-6c65-4257-9a2d-d66ce12fcde2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.970307ms', 'executionTime': '62.903828ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:40:56,070] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:40:56,096] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:40:56,120] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:40:56,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 668.841894ms
[2022-09-02 03:40:56,791] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:40:56,845] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:40:56,883] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:40:56,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.392064ms
[2022-09-02 03:40:57,172] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:57,238] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:40:57,256] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:40:57,257] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:40:57,274] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:40:57,344] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:58,081] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeaead9de0b30c4b6f85b3617c98c7c16djob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:40:58,107] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:40:58,142] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.745217ms
[2022-09-02 03:40:58,143] - [base_gsi:282] INFO - BUILD INDEX on default(employeeaead9de0b30c4b6f85b3617c98c7c16djob_title) USING GSI
[2022-09-02 03:40:59,174] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeaead9de0b30c4b6f85b3617c98c7c16djob_title) USING GSI
[2022-09-02 03:40:59,200] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%29+USING+GSI
[2022-09-02 03:40:59,226] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.324009ms
[2022-09-02 03:41:00,258] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:00,285] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:00,293] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.916549ms
[2022-09-02 03:41:01,325] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:01,352] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:01,360] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.828322ms
[2022-09-02 03:41:01,875] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:41:02,054] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:03,151] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:03,835] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-02 03:41:04,043] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:05,203] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:05,598] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:41:05,776] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:09,512] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:10,382] - [task:3235] INFO -  <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:10,414] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:41:10,441] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:41:10,446] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.041758ms
[2022-09-02 03:41:10,446] - [task:3245] INFO - {'requestID': '0f630634-2747-4422-9b45-e09de310c682', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title', 'index_id': '99b553725d16e643', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE  job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.041758ms', 'executionTime': '2.974578ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:41:10,446] - [task:3246] INFO -  <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:10,447] - [task:3276] INFO -  <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:10,447] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:41:10,447] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:41:10,448] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["job_title"] == "Sales" 
[2022-09-02 03:41:10,448] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:41:10,448] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:41:10,448] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:41:10,449] - [tuq_generators:329] INFO - -->where_clause=  doc["job_title"] == "Sales" 
[2022-09-02 03:41:11,448] - [task:3235] INFO -  <<<<< START Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:11,478] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  job_title = "Sales" 
[2022-09-02 03:41:11,504] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:41:11,660] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 141.911199ms
[2022-09-02 03:41:11,661] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:41:11,661] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:41:13,301] - [task:3246] INFO -  <<<<< Done Executing Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:13,301] - [task:3276] INFO -  <<<<< Done VERIFYING Query SELECT * FROM default WHERE  job_title = "Sales"  >>>>>>
[2022-09-02 03:41:14,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:14,356] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:14,363] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.746245ms
[2022-09-02 03:41:14,389] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeaead9de0b30c4b6f85b3617c98c7c16djob_title ON default USING GSI
[2022-09-02 03:41:14,415] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeaead9de0b30c4b6f85b3617c98c7c16djob_title+ON+default+USING+GSI
[2022-09-02 03:41:14,461] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.590207ms
[2022-09-02 03:41:14,494] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:14,520] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:14,527] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.968596ms
[2022-09-02 03:41:14,527] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'ade83831-a589-4511-b20c-92439a99d474', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.968596ms', 'executionTime': '5.896591ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:41:14,631] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:14,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:14,635] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:15,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:15,339] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:41:15,339] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:41:15,444] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:41:15,503] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:15,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:15,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:16,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:16,222] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:41:16,222] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:41:16,327] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:41:16,382] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:16,383] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:41:16,383] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:41:16,408] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:41:16,433] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:41:16,441] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.161441ms
[2022-09-02 03:41:16,467] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:41:16,493] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:41:16,545] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.597112ms
[2022-09-02 03:41:16,612] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:41:16,612] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.75264437955159, 'mem_free': 12850675712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:41:16,613] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:41:16,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:16,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:17,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:17,380] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:17,380] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:17,998] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:18,003] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:18,003] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:18,804] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:18,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:18,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:19,928] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:27,736] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:41:27,878] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:41:28,380] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:41:28,410] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:41:28,411] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:41:28,470] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,525] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,578] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,579] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:41:28,656] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:41:28,657] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:41:28,682] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,810] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:41:28,810] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,837] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:41:28,863] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:41:28,863] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,890] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:41:28,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:41:28,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,942] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:41:28,968] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:41:28,969] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:29,000] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:41:29,001] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:41:29,001] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:41:29,002] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6
ok

----------------------------------------------------------------------
Ran 1 test in 128.032s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7'}
[2022-09-02 03:41:29,145] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:29,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:29,728] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:29,761] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:29,841] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:41:29,841] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-02 03:41:29,841] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:41:30,172] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:41:30,202] - [task:164] INFO -  {'uptime': '899', 'memoryTotal': 15466930176, 'memoryFree': 13087059968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:30,228] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:41:30,229] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:41:30,229] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:41:30,259] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:41:30,295] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:41:30,295] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:41:30,327] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:41:30,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:30,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:30,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:30,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:30,987] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:31,100] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:31,101] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:31,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:31,133] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:31,268] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:41:31,295] - [task:164] INFO -  {'uptime': '897', 'memoryTotal': 15466930176, 'memoryFree': 13087145984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:31,321] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:31,348] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:41:31,349] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:41:31,401] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:31,404] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:31,405] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:31,996] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:31,997] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:32,111] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:32,112] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:32,142] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:32,143] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:32,263] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:41:32,290] - [task:164] INFO -  {'uptime': '898', 'memoryTotal': 15466930176, 'memoryFree': 13086478336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:32,315] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:32,343] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:41:32,343] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:41:32,397] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:32,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:32,403] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:33,021] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:33,022] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:33,137] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:33,139] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:33,169] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:33,170] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:33,287] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:41:33,313] - [task:164] INFO -  {'uptime': '897', 'memoryTotal': 15466930176, 'memoryFree': 13086650368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:33,339] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:33,366] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:41:33,367] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:41:33,425] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:33,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:33,428] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:34,027] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:34,028] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:34,143] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:34,144] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:34,173] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:34,174] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:34,267] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:41:34,695] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:41:39,700] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:41:39,783] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:41:39,797] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:39,797] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:40,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:40,393] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:40,517] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:40,518] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:40,518] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:41:41,401] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:41:41,454] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:41:41,455] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:42:05,763] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:06,074] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:06,333] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:42:06,336] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:42:06,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:06,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:06,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:06,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:06,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:07,762] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:07,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:07,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:08,811] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:08,821] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:08,821] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:09,868] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:16,175] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:42:16,176] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.80845304426461, 'mem_free': 13098573824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:42:16,176] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:42:16,176] - [basetestcase:467] INFO - Time to execute basesetup : 47.03368258476257
[2022-09-02 03:42:16,229] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,229] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:42:16,281] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,282] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:42:16,336] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,337] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:42:16,389] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,389] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:42:16,441] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:16,501] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:42:16,501] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:42:16,502] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:42:21,513] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:42:21,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:21,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:22,092] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:23,237] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:42:23,405] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:25,590] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:42:25,670] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:25,670] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:42:25,670] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:42:55,693] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:42:55,724] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:42:55,749] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:42:55,815] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.386652ms
[2022-09-02 03:42:55,815] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9d101b02-3d1b-42c7-bb1b-d0cc107937c3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.386652ms', 'executionTime': '63.326441ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:42:55,815] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:42:55,841] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:42:55,866] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:42:56,513] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 644.552306ms
[2022-09-02 03:42:56,513] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:42:56,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:42:56,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:42:56,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.370373ms
[2022-09-02 03:42:56,817] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:56,855] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:42:56,872] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:42:56,873] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:42:56,889] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:42:56,890] - [base_gsi:326] INFO - []
[2022-09-02 03:42:57,781] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title` ON default(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:42:57,807] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:42:57,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.201279ms
[2022-09-02 03:42:57,876] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:42:57,901] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:42:57,942] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.638628ms
[2022-09-02 03:42:57,944] - [base_gsi:282] INFO - BUILD INDEX on default(employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title,employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr) USING GSI
[2022-09-02 03:42:58,976] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title,employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr) USING GSI
[2022-09-02 03:42:59,001] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%2Cemployeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%29+USING+GSI
[2022-09-02 03:42:59,036] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.480995ms
[2022-09-02 03:43:00,067] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:00,094] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:00,103] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.437035ms
[2022-09-02 03:43:01,136] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:01,164] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:01,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.364551ms
[2022-09-02 03:43:01,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:01,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:01,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.763417ms
[2022-09-02 03:43:02,260] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:02,286] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:02,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.65221ms
[2022-09-02 03:43:02,321] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title ON default USING GSI
[2022-09-02 03:43:02,346] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title+ON+default+USING+GSI
[2022-09-02 03:43:02,401] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.162018ms
[2022-09-02 03:43:02,435] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:02,461] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:02,469] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.367097ms
[2022-09-02 03:43:02,495] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr ON default USING GSI
[2022-09-02 03:43:02,524] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr+ON+default+USING+GSI
[2022-09-02 03:43:02,562] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.884846ms
[2022-09-02 03:43:02,598] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:02,623] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:02,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.097315ms
[2022-09-02 03:43:02,630] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '12a42918-556b-4784-b452-db67c4cb01e8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.097315ms', 'executionTime': '5.029226ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:43:02,655] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:02,680] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:02,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.169466ms
[2022-09-02 03:43:02,683] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2ede51e4-5846-4aa9-b1f2-2ff9a37c5671', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.169466ms', 'executionTime': '1.11794ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:43:02,786] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:02,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:02,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:03,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:03,435] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:43:03,435] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:43:03,533] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:43:03,587] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:03,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:03,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:04,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:04,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:43:04,267] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:43:04,372] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:43:04,430] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:04,430] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:43:04,430] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:43:04,457] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:43:04,483] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:43:04,491] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.270669ms
[2022-09-02 03:43:04,516] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:43:04,542] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:43:04,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.883249ms
[2022-09-02 03:43:04,655] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:43:04,656] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.9021964191466, 'mem_free': 12968845312, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:43:04,656] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:43:04,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:04,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:05,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:05,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:05,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:05,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:05,904] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:05,904] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:06,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:06,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:06,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:07,760] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:14,851] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:43:15,004] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:43:15,417] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:43:15,445] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:43:15,445] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:43:15,503] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,556] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,609] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,610] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:43:15,689] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:43:15,690] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:43:15,715] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,844] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:43:15,844] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,870] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:43:15,897] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:43:15,897] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,925] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:43:15,951] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:43:15,952] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,979] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:43:16,005] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:43:16,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:16,034] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:43:16,034] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:43:16,034] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:43:16,035] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7
ok

----------------------------------------------------------------------
Ran 1 test in 106.949s

OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8'}
[2022-09-02 03:43:16,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:16,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:16,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:16,804] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:16,884] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:43:16,884] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-02 03:43:16,885] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:43:17,158] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:43:17,188] - [task:164] INFO -  {'uptime': '1006', 'memoryTotal': 15466930176, 'memoryFree': 13097017344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:17,216] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:43:17,216] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:43:17,217] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:43:17,248] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:43:17,294] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:43:17,294] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:43:17,322] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:43:17,323] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:43:17,324] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:43:17,324] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:43:17,372] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:17,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:17,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:17,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:17,989] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:18,105] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:18,106] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:18,136] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:18,136] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:18,256] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:43:18,283] - [task:164] INFO -  {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13097246720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:18,308] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:18,337] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:43:18,337] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:43:18,388] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:18,391] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:18,391] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:18,979] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:18,980] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:19,100] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:19,101] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:19,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:19,132] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:19,250] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:43:19,277] - [task:164] INFO -  {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13097013248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:19,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:19,330] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:43:19,331] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:43:19,381] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:19,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:19,386] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:19,976] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:19,977] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:20,094] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:20,095] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:20,130] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:20,131] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:20,247] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:43:20,274] - [task:164] INFO -  {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13096919040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:20,301] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:20,328] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:43:20,328] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:43:20,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:20,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:20,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:20,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:20,989] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:21,103] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:21,104] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:21,136] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:21,137] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:21,231] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:43:21,628] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:43:26,634] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:43:26,717] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:43:26,722] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:26,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:27,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:27,337] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:27,450] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:27,451] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:27,451] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:43:28,346] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:43:28,401] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:43:28,402] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:44:05,709] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:06,022] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:06,374] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:44:06,377] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:44:06,437] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:06,437] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:07,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:07,043] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:07,043] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:07,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:07,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:07,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:09,002] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:09,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:09,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:10,078] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:16,538] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:44:16,539] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.44980424605788, 'mem_free': 13095411712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:44:16,539] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:44:16,539] - [basetestcase:467] INFO - Time to execute basesetup : 60.41486215591431
[2022-09-02 03:44:16,590] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,590] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:44:16,646] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,646] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:44:16,702] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,702] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:44:16,758] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,758] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:44:16,813] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:16,875] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:44:16,876] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:44:16,876] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:44:21,884] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:44:21,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:21,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:22,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:23,555] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:44:23,722] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:25,955] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:44:26,036] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:26,036] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:44:26,036] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:44:56,066] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:44:56,095] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:44:56,120] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:44:56,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.89703ms
[2022-09-02 03:44:56,184] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'aee697de-7379-4eeb-bda1-0bb835f46e7b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '61.89703ms', 'executionTime': '61.833391ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:44:56,185] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:44:56,211] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:44:56,236] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:44:56,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 638.631252ms
[2022-09-02 03:44:56,877] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:44:56,951] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:44:56,994] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:44:57,006] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.918848ms
[2022-09-02 03:44:57,247] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:57,283] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:44:57,311] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:44:57,312] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:44:57,337] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:44:57,338] - [base_gsi:326] INFO - []
[2022-09-02 03:44:58,213] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee09c2890e7403479a81167a6a98ff8403job_title_join_yr` ON default(join_yr,job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:44:58,238] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:44:58,280] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.570872ms
[2022-09-02 03:44:58,281] - [base_gsi:282] INFO - BUILD INDEX on default(employee09c2890e7403479a81167a6a98ff8403job_title_join_yr) USING GSI
[2022-09-02 03:44:59,311] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee09c2890e7403479a81167a6a98ff8403job_title_join_yr) USING GSI
[2022-09-02 03:44:59,337] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%29+USING+GSI
[2022-09-02 03:44:59,360] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.535152ms
[2022-09-02 03:45:00,392] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:00,418] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:00,427] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.561603ms
[2022-09-02 03:45:01,459] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:01,487] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:01,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.892542ms
[2022-09-02 03:45:02,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:02,555] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:02,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.447795ms
[2022-09-02 03:45:02,593] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee09c2890e7403479a81167a6a98ff8403job_title_join_yr ON default USING GSI
[2022-09-02 03:45:02,621] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee09c2890e7403479a81167a6a98ff8403job_title_join_yr+ON+default+USING+GSI
[2022-09-02 03:45:02,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.529735ms
[2022-09-02 03:45:02,704] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:02,735] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:02,742] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.994653ms
[2022-09-02 03:45:02,743] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2a562b68-8aca-4dde-bdf8-173679fed5da', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.994653ms', 'executionTime': '5.933741ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:45:02,852] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:02,856] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:02,856] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:03,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:03,541] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:45:03,541] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:45:03,641] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:45:03,696] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:03,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:03,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:04,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:04,382] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:45:04,383] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:45:04,484] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:45:04,540] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:04,541] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:45:04,541] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:45:04,567] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:45:04,593] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:45:04,600] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.4227ms
[2022-09-02 03:45:04,625] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:45:04,650] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:45:04,704] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.037635ms
[2022-09-02 03:45:04,775] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:45:04,775] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 11.91682580504829, 'mem_free': 12939608064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:45:04,775] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:45:04,782] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:04,782] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:05,410] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:05,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:05,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:06,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:06,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:06,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:07,558] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:07,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:07,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:08,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:15,333] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:45:15,478] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:45:16,314] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:45:16,342] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:45:16,342] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:45:16,397] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,450] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,505] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,506] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:45:16,587] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:45:16,590] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:45:16,627] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,760] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:45:16,761] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,787] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:45:16,813] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:45:16,813] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,841] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:45:16,867] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:45:16,867] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,893] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:45:16,918] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:45:16,919] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,945] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:45:16,946] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:45:16,946] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:45:16,947] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8
ok

----------------------------------------------------------------------
Ran 1 test in 120.876s

OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9'}
[2022-09-02 03:45:17,036] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:17,036] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:17,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:17,683] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:17,762] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:45:17,762] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-02 03:45:17,763] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:45:18,062] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:45:18,095] - [task:164] INFO -  {'uptime': '1127', 'memoryTotal': 15466930176, 'memoryFree': 13090742272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:18,121] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:45:18,121] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:45:18,121] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:45:18,150] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:45:18,187] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:45:18,188] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:45:18,216] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:45:18,266] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:18,269] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:18,269] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:18,873] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:18,874] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:18,992] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:18,993] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:19,024] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:19,025] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:19,148] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:45:19,175] - [task:164] INFO -  {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091569664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:19,200] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:19,228] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:45:19,228] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:45:19,279] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:19,282] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:19,283] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:19,897] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:19,898] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:20,014] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:20,016] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:20,049] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:20,050] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:20,167] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:45:20,195] - [task:164] INFO -  {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091950592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:20,220] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:20,248] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:45:20,248] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:45:20,300] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:20,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:20,306] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:20,929] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:20,930] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:21,052] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:21,053] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:21,083] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:21,084] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:21,201] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:45:21,227] - [task:164] INFO -  {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091540992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:21,253] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:21,281] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:45:21,282] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:45:21,337] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:21,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:21,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:21,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:21,942] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:22,062] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:22,063] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:22,093] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:22,093] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:22,185] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:45:22,578] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:45:27,582] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:45:27,666] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:45:27,671] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:27,674] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:28,302] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:28,304] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:28,423] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:28,424] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:28,424] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:45:29,307] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:45:29,363] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:45:29,363] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:46:05,612] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:05,939] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:06,208] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:46:06,212] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:46:06,278] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:06,278] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:06,981] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:06,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:06,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:07,954] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:07,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:07,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:09,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:09,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:09,070] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:10,166] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:16,708] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:46:16,710] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.17919900590499, 'mem_free': 13086642176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:46:16,711] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:46:16,711] - [basetestcase:467] INFO - Time to execute basesetup : 59.67742037773132
[2022-09-02 03:46:16,762] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,763] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:46:16,817] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,817] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:46:16,870] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,871] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:46:16,928] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,928] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:46:16,983] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:17,047] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:46:17,047] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:46:17,047] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:46:22,058] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:46:22,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:22,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:22,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:23,858] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:46:24,049] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:27,176] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:46:27,256] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:27,256] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:46:27,257] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:46:57,286] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:46:57,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:46:57,352] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:46:57,424] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.478348ms
[2022-09-02 03:46:57,424] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '18739bd7-b20f-4dd1-a13c-d7cc46710ab1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.478348ms', 'executionTime': '69.399192ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:46:57,424] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:46:57,452] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:46:57,479] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:46:58,163] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.9479ms
[2022-09-02 03:46:58,163] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:46:58,216] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:46:58,256] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:46:58,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.414397ms
[2022-09-02 03:46:58,480] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:58,525] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:46:58,555] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:46:58,555] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:46:58,571] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:46:58,642] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:59,440] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4b6c1deee4d047558d38b2d6c2e111a3join_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:46:59,465] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:46:59,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.224513ms
[2022-09-02 03:46:59,496] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4b6c1deee4d047558d38b2d6c2e111a3join_yr`) USING GSI
[2022-09-02 03:47:00,528] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4b6c1deee4d047558d38b2d6c2e111a3join_yr`) USING GSI
[2022-09-02 03:47:00,555] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%60%29+USING+GSI
[2022-09-02 03:47:00,581] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.520428ms
[2022-09-02 03:47:00,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:00,663] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:00,672] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.610843ms
[2022-09-02 03:47:01,705] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:01,732] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:01,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.933808ms
[2022-09-02 03:47:02,778] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:02,803] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:02,810] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.856132ms
[2022-09-02 03:47:03,841] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:47:03,868] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:47:03,872] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.890287ms
[2022-09-02 03:47:03,872] - [base_gsi:504] INFO - {'requestID': 'a92cb88c-8fbd-4f81-8fcc-ac9b19b69abd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr', 'index_id': '9da2e12ce9deead3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.890287ms', 'executionTime': '2.831206ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:47:03,873] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:47:03,873] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:47:03,874] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:47:03,874] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:47:03,875] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,875] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,876] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:47:03,876] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:47:03,935] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,979] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:47:04,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:47:04,190] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.079145ms
[2022-09-02 03:47:04,190] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:47:04,191] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:47:06,465] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:06,495] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:06,541] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:06,547] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.778306ms
[2022-09-02 03:47:06,547] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '183cab59-02dc-43fe-85fb-5f5a2c485b99', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.778306ms', 'executionTime': '4.718788ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:47:06,657] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:06,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:06,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:07,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:07,489] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:47:07,490] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:47:07,602] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:47:07,659] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:07,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:07,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:08,356] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:08,415] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:47:08,415] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:47:08,529] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:47:08,593] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:08,593] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:47:08,593] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:47:08,620] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:47:08,646] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:47:08,653] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.814265ms
[2022-09-02 03:47:08,653] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '1206497e-c4d9-4a07-a312-9e24a27a146f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.814265ms', 'executionTime': '5.754035ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:47:08,705] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:47:08,706] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.581473987757392, 'mem_free': 12942626816, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:47:08,706] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:47:08,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:08,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:09,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:09,364] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:09,364] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:10,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:10,250] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:10,250] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:11,436] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:11,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:11,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:12,624] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:19,845] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:47:19,986] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,040] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,093] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,146] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,147] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:47:20,224] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:47:20,225] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:47:20,250] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,381] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:47:20,381] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,408] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:47:20,434] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:47:20,434] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,460] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:47:20,486] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:47:20,486] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,512] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:47:20,539] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:47:20,539] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,567] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:47:20,567] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:47:20,568] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:47:20,568] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9
ok

----------------------------------------------------------------------
Ran 1 test in 123.591s

OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10'}
[2022-09-02 03:47:20,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:20,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:21,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:21,402] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:21,486] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:47:21,486] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-02 03:47:21,487] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:47:21,687] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:47:21,713] - [task:164] INFO -  {'uptime': '1252', 'memoryTotal': 15466930176, 'memoryFree': 13096235008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:21,739] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:47:21,739] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:47:21,740] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:47:21,775] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:47:21,809] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:47:21,810] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:47:21,838] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:47:21,888] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:21,891] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:21,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:22,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:22,534] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:22,655] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:22,656] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:22,686] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:22,686] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:22,814] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:47:22,841] - [task:164] INFO -  {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096726528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:22,869] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:22,898] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:47:22,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:47:22,950] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:22,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:22,957] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:23,594] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:23,596] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:23,721] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:23,722] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:23,751] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:23,752] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:23,868] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:47:23,895] - [task:164] INFO -  {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096243200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:23,921] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:23,950] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:47:23,950] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:47:24,005] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:24,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:24,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:24,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:24,694] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:24,826] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:24,827] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:24,861] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:24,862] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:24,990] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:47:25,020] - [task:164] INFO -  {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096681472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:25,046] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:25,075] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:47:25,076] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:47:25,130] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:25,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:25,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:25,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:25,824] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:25,956] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:25,957] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:25,987] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:25,988] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:26,080] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:47:26,455] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:47:31,461] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:47:31,556] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:47:31,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:31,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:32,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:32,249] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:32,382] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:32,383] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:32,383] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:47:33,241] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:47:33,297] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:47:33,298] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:48:05,840] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:06,155] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:06,443] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:48:06,447] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-02 03:48:06,512] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:06,512] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:07,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:07,167] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:07,167] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:08,161] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:08,179] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:08,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:09,339] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:09,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:09,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:10,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:17,462] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:48:17,463] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.68369701692233, 'mem_free': 13094109184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:48:17,463] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:48:17,464] - [basetestcase:467] INFO - Time to execute basesetup : 56.80632734298706
[2022-09-02 03:48:17,518] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,518] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:48:17,574] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,574] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:48:17,628] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,628] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:48:17,685] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,685] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:48:17,739] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:17,807] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:48:17,807] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:48:17,807] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:48:22,820] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:48:22,823] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:22,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:23,527] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:24,731] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:48:24,905] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:27,945] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:48:28,028] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:28,028] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:48:28,029] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:48:58,055] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:48:58,086] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:48:58,111] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:48:58,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.220952ms
[2022-09-02 03:48:58,177] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '97ec36e0-2b17-40ca-9a6d-9250e7dab5ca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.220952ms', 'executionTime': '63.156224ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:48:58,177] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:48:58,202] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:48:58,226] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:48:58,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 627.0798ms
[2022-09-02 03:48:58,855] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:48:58,925] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:48:58,960] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:48:58,970] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.560628ms
[2022-09-02 03:48:59,198] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:59,234] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:48:59,249] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:48:59,250] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:48:59,262] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:48:59,350] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:00,160] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6ec54c502be04606bd31701bd27b3a5ejoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:49:00,186] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:49:00,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.381888ms
[2022-09-02 03:49:00,221] - [base_gsi:282] INFO - BUILD INDEX on default(`employee6ec54c502be04606bd31701bd27b3a5ejoin_yr`) USING GSI
[2022-09-02 03:49:01,252] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee6ec54c502be04606bd31701bd27b3a5ejoin_yr`) USING GSI
[2022-09-02 03:49:01,278] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%60%29+USING+GSI
[2022-09-02 03:49:01,308] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.004146ms
[2022-09-02 03:49:01,349] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:01,385] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:01,394] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.097798ms
[2022-09-02 03:49:02,424] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:02,450] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:02,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.264989ms
[2022-09-02 03:49:03,489] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:03,515] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:03,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.661173ms
[2022-09-02 03:49:04,555] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:49:04,584] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:49:04,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.872032ms
[2022-09-02 03:49:04,588] - [base_gsi:504] INFO - {'requestID': '20ea87d2-831f-48cb-a402-0ec19f6f57c0', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr', 'index_id': '477ca682526371b0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.872032ms', 'executionTime': '2.797905ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:49:04,589] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:49:04,589] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:04,590] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:49:04,590] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:49:04,590] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,590] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,591] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:04,591] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:04,652] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,693] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:49:04,719] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:49:04,889] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 158.301809ms
[2022-09-02 03:49:04,890] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:49:04,890] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:49:06,136] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param: 
[2022-09-02 03:49:06,194] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-02 03:49:06,224] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:49:06,251] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:49:06,254] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.875253ms
[2022-09-02 03:49:06,254] - [base_gsi:504] INFO - {'requestID': 'c474215e-ea15-4e65-ba28-891de89aed5b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr', 'index_id': '477ca682526371b0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.875253ms', 'executionTime': '1.813943ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:49:06,255] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:49:06,255] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:06,255] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:49:06,255] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:49:06,256] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,256] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,256] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:06,256] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:49:06,308] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,346] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:49:06,371] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:49:06,463] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 83.048508ms
[2022-09-02 03:49:06,463] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:49:06,464] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:49:07,741] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee6ec54c502be04606bd31701bd27b3a5ejoin_yr ON default USING GSI
[2022-09-02 03:49:07,768] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee6ec54c502be04606bd31701bd27b3a5ejoin_yr+ON+default+USING+GSI
[2022-09-02 03:49:07,814] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.725086ms
[2022-09-02 03:49:07,848] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:07,877] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:07,884] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.484406ms
[2022-09-02 03:49:07,884] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '9df9d843-3f98-4a3b-981c-7aa21335d831', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.484406ms', 'executionTime': '5.423583ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:49:07,988] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:07,991] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:07,991] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:08,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:08,740] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:49:08,740] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:49:08,857] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:49:08,917] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:08,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:08,920] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:09,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:09,719] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:49:09,719] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:49:09,836] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:49:09,893] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:09,893] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:49:09,893] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:49:09,919] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:49:09,945] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:49:09,952] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.557482ms
[2022-09-02 03:49:09,977] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:49:10,003] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:49:10,051] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.389252ms
[2022-09-02 03:49:10,119] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:49:10,119] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.83201563203367, 'mem_free': 12944936960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:49:10,120] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:49:10,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:10,124] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:10,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:10,826] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:10,826] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:11,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:11,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:11,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:12,557] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:12,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:12,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:13,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:21,566] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-02 03:49:21,712] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:49:22,359] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:49:22,387] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:49:22,387] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:49:22,441] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,493] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,546] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,547] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:49:22,626] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:49:22,627] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:49:22,653] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,794] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:49:22,794] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,822] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:49:22,860] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:49:22,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,887] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:49:22,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:49:22,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,943] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:49:22,969] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:49:22,969] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,998] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:49:22,998] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-02 03:49:22,998] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:49:22,999] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10
ok

----------------------------------------------------------------------
Ran 1 test in 122.396s

OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11

./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi

Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11'}
[2022-09-02 03:49:23,089] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:23,089] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:23,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:23,787] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:23,867] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:49:23,867] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-02 03:49:23,868] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:49:24,120] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:49:24,147] - [task:164] INFO -  {'uptime': '1373', 'memoryTotal': 15466930176, 'memoryFree': 13081886720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:24,173] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:49:24,174] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:49:24,174] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:49:24,207] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:49:24,243] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:49:24,244] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:49:24,271] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:49:24,272] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:49:24,272] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:49:24,273] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:49:24,324] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:24,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:24,328] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:25,001] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:25,002] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:25,132] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:25,133] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:25,163] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:25,163] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:25,288] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:49:25,320] - [task:164] INFO -  {'uptime': '1369', 'memoryTotal': 15466930176, 'memoryFree': 13081690112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:25,347] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:25,376] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:49:25,377] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:49:25,432] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:25,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:25,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:26,164] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:26,165] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:26,307] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:26,308] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:26,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:26,342] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:26,470] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:49:26,499] - [task:164] INFO -  {'uptime': '1370', 'memoryTotal': 15466930176, 'memoryFree': 13082599424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:26,528] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:26,557] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:49:26,558] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:49:26,611] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:26,617] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:26,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:27,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:27,341] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:27,484] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:27,485] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:27,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:27,519] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:27,640] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:49:27,667] - [task:164] INFO -  {'uptime': '1369', 'memoryTotal': 15466930176, 'memoryFree': 13082394624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:27,695] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:27,724] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:49:27,725] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:49:27,782] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:27,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:27,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:28,472] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:28,473] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:28,600] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:28,601] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:28,632] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:28,632] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:28,723] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:49:29,105] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:49:34,110] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:49:34,201] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:49:34,207] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:34,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:34,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:34,908] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:35,043] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:35,044] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:35,045] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:49:35,902] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:49:35,960] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:49:35,960] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:50:05,766] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:06,079] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:06,341] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:50:06,344] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:50:06,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:06,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:07,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:07,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:07,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:08,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:08,182] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:08,182] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:09,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:09,422] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:09,422] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:10,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:17,888] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:50:17,889] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.73216744020807, 'mem_free': 13078851584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:50:17,889] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:50:17,889] - [basetestcase:467] INFO - Time to execute basesetup : 54.80256128311157
[2022-09-02 03:50:17,944] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:17,944] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:50:18,001] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,001] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:50:18,054] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,054] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:50:18,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,109] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:50:18,161] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:18,223] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:50:18,223] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:50:18,224] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:50:23,236] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:50:23,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:23,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:23,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:25,186] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:50:25,374] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:28,330] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:50:28,412] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:28,413] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:50:28,413] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:50:58,439] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:50:58,468] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:50:58,493] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:50:58,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.929771ms
[2022-09-02 03:50:58,559] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '52b5abb4-6cad-4518-8f52-7ee3e81a3160', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.929771ms', 'executionTime': '63.859059ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:50:58,559] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:50:58,585] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:50:58,610] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:50:59,308] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 696.545099ms
[2022-09-02 03:50:59,309] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:50:59,365] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:50:59,402] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:50:59,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.129552ms
[2022-09-02 03:50:59,614] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:59,651] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:50:59,671] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:50:59,674] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:50:59,693] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:50:59,772] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:51:00,573] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr` ON default(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'defer_build': True}
[2022-09-02 03:51:00,599] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:51:00,641] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.226025ms
[2022-09-02 03:51:00,641] - [base_gsi:282] INFO - BUILD INDEX on default(`employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr`) USING GSI
[2022-09-02 03:51:01,673] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr`) USING GSI
[2022-09-02 03:51:01,701] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%60%29+USING+GSI
[2022-09-02 03:51:01,724] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.175858ms
[2022-09-02 03:51:01,780] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:01,811] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:01,820] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.523242ms
[2022-09-02 03:51:02,851] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:02,876] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:02,883] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.610763ms
[2022-09-02 03:51:03,914] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:03,942] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:03,951] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.667847ms
[2022-09-02 03:51:04,982] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:51:05,007] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:51:05,012] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.591706ms
[2022-09-02 03:51:05,013] - [base_gsi:504] INFO - {'requestID': '021939d3-bbe2-4633-a6ec-e6d323cd140e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr', 'index_id': '3064d7a6164a52a4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.591706ms', 'executionTime': '3.524317ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:51:05,013] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:51:05,013] - [tuq_generators:72] INFO - WHERE clause ===== is   doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:51:05,014] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:51:05,014] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:51:05,014] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,014] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,015] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:51:05,015] - [tuq_generators:329] INFO - -->where_clause=  doc["join_yr"] > 2010 and doc["join_yr"] < 2014 
[2022-09-02 03:51:05,074] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id 
[2022-09-02 03:51:05,138] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:51:05,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.385662ms
[2022-09-02 03:51:05,323] - [tuq_helper:409] INFO -  Analyzing Actual Result
[2022-09-02 03:51:05,324] - [tuq_helper:411] INFO -  Analyzing Expected Result
[2022-09-02 03:51:07,325] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-02 03:51:10,062] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:51:10,124] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:51:10,124] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:52:05,569] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:52:05,888] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:52:06,288] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:52:06,420] - [basetestcase:847] INFO - sleep for 2 secs.  ...
[2022-09-02 03:52:08,457] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:52:08,458] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,487] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:52:08,517] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:52:08,517] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,545] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:52:08,574] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:52:08,575] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,603] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:52:08,631] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:52:08,631] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,658] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:52:08,710] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:08,736] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:52:08,762] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:52:08,804] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.436456ms
[2022-09-02 03:52:08,804] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'b1963f3d-d268-4e4e-aeeb-a84a569125d9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '40.436456ms', 'executionTime': '40.377327ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:08,832] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:52:08,858] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:52:08,861] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.082086ms
[2022-09-02 03:52:08,861] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '8d5315ac-46f9-431e-a51e-c3d7d4549d30', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.082086ms', 'executionTime': '1.02148ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:08,916] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:09,046] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:09,047] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:09,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:09,766] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:09,827] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:52:09,828] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:52:09,942] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:52:09,998] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:10,001] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:10,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:10,727] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:10,785] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:52:10,786] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:52:10,906] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:52:10,967] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:10,967] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:52:10,967] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:52:10,993] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:52:11,020] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:52:11,027] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.1186ms
[2022-09-02 03:52:11,027] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '2bd56144-eeb5-4825-9bf5-c42b42a83f45', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.1186ms', 'executionTime': '5.04852ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:11,027] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:52:11,055] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:52:11,083] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:52:11,086] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.381899ms
[2022-09-02 03:52:11,087] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': '6299aac7-ea6f-4fe4-9a30-c6d8a98f07f5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.381899ms', 'executionTime': '1.312904ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:11,146] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:52:11,146] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.060210697253556, 'mem_free': 13067821056, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:52:11,147] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:52:11,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:11,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:11,876] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:11,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:11,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:12,797] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:12,809] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:12,810] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:14,055] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:14,067] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:14,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:15,285] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:22,994] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:52:23,146] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:52:23,464] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:52:23,494] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:52:23,494] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:52:23,552] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,663] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,664] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:52:23,750] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:52:23,751] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:52:23,778] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:23,917] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:52:23,917] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:23,947] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:52:23,975] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:52:23,975] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,004] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:52:24,030] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:52:24,030] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,059] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:52:24,088] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:52:24,088] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,118] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:52:24,118] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:52:24,119] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:52:24,119] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 181.091s

OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11

*** Tests executed count: 11

Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-02 03:52:24,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:24,185] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:24,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:24,996] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:25,079] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:52:25,080] - [basetestcase:157] INFO - ==============  basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-02 03:52:25,080] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:52:25,213] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self 
[2022-09-02 03:52:25,242] - [task:164] INFO -  {'uptime': '1554', 'memoryTotal': 15466930176, 'memoryFree': 13065805824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:25,270] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:52:25,270] - [task:205] INFO - set index quota to node 127.0.0.1 
[2022-09-02 03:52:25,271] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:52:25,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:52:25,343] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:52:25,344] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:52:25,372] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:52:25,373] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:52:25,373] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:52:25,374] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:52:25,422] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:25,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:25,426] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:26,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:26,136] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:26,271] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:26,273] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:26,305] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:26,306] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:26,425] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self 
[2022-09-02 03:52:26,453] - [task:164] INFO -  {'uptime': '1555', 'memoryTotal': 15466930176, 'memoryFree': 13066162176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:26,479] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:26,507] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:52:26,508] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:52:26,560] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:26,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:26,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:27,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:27,291] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:27,421] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:27,422] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:27,453] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:27,454] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:27,572] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self 
[2022-09-02 03:52:27,599] - [task:164] INFO -  {'uptime': '1550', 'memoryTotal': 15466930176, 'memoryFree': 13064998912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:27,625] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:27,653] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:52:27,654] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:52:27,706] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:27,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:27,709] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:28,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:28,443] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:28,588] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:28,589] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:28,620] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:28,621] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:28,746] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self 
[2022-09-02 03:52:28,776] - [task:164] INFO -  {'uptime': '1550', 'memoryTotal': 15466930176, 'memoryFree': 13058134016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:28,803] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:28,834] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:52:28,834] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:52:28,887] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:28,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:28,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:29,587] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:29,588] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:29,728] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:29,729] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:29,760] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:29,761] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:29,864] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:52:30,258] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:52:35,260] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:52:35,346] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:52:35,350] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:35,350] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:36,052] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:36,053] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:36,194] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:36,195] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:36,196] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:52:37,045] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:52:37,104] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:52:37,105] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:53:05,846] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:06,171] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:06,454] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:53:06,457] - [basetestcase:435] INFO - ==============  basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-02 03:53:06,515] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:06,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:07,266] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:07,271] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:07,271] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:08,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:08,460] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:08,461] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:09,728] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:09,736] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:09,736] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:11,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:18,460] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:53:18,460] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.15422214916433, 'mem_free': 13059473408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:53:18,460] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:53:18,461] - [basetestcase:467] INFO - Time to execute basesetup : 54.27910399436951
[2022-09-02 03:53:18,513] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,513] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:53:18,571] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,571] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:53:18,625] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,625] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:53:18,679] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,680] - [newtuq:28] INFO - current status of 127.0.0.1  is healthy
[2022-09-02 03:53:18,735] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:53:18,792] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:53:18,793] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:53:18,793] - [basetestcase:847] INFO - sleep for 5 secs.  ...
[2022-09-02 03:53:23,805] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:53:23,809] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:23,809] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:24,515] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:25,610] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:53:25,780] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:28,874] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:53:28,951] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:53:28,951] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:53:28,952] - [basetestcase:847] INFO - sleep for 30 secs.  ...
[2022-09-02 03:53:58,986] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:53:59,016] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:53:59,042] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:53:59,112] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.255591ms
[2022-09-02 03:53:59,113] - [tuq_helper:919] ERROR - Fail to get index list.  List output: {'requestID': 'db60f5c0-5c8e-4943-8f77-5fc6e2186805', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.255591ms', 'executionTime': '68.193083ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:53:59,113] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:53:59,139] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default 
[2022-09-02 03:53:59,165] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:53:59,837] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 670.40011ms
[2022-09-02 03:53:59,838] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:53:59,913] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:53:59,946] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:53:59,953] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.874546ms
[2022-09-02 03:54:00,172] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,215] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:54:00,250] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:54:00,251] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:54:00,269] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:54:00,331] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,464] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,467] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:00,467] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:01,307] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:01,365] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:54:01,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:54:01,494] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:54:01,551] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:01,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:01,554] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:02,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:02,436] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:54:02,436] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:54:02,578] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:54:02,635] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:02,635] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:54:02,635] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:54:02,660] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:54:02,688] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:54:02,695] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.782303ms
[2022-09-02 03:54:02,723] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:54:02,752] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:54:02,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.686858ms
[2022-09-02 03:54:02,864] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:54:02,864] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.635379814318155, 'mem_free': 12891725824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:54:02,864] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:54:02,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:02,869] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:03,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:03,655] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:03,655] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:04,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:04,464] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:04,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:05,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:05,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:05,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:07,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:15,919] - [basetestcase:719] INFO - ==============  basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-02 03:54:16,285] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:54:17,390] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:54:17,430] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:54:17,430] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:54:17,489] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,544] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,603] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,604] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:54:17,684] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:54:17,685] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:54:17,714] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,846] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:54:17,846] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,874] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:54:17,901] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:54:17,901] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,929] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:54:17,955] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:54:17,955] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,982] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:54:18,008] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:54:18,008] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:18,035] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:54:18,036] - [basetestcase:761] INFO - ==============  basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-02 03:54:18,036] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:54:18,036] - [basetestcase:778] INFO - closing all memcached connections
ok

----------------------------------------------------------------------
Ran 1 test in 113.929s

OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91:  6055 Terminated              COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log  (wd: /opt/build/ns_server)

Testing Failed: Required test failed

Binary file /opt/build/test.log matches
Binary file /opt/build/test.log matches
Version: versions-02.09.2022-01.46.cfg
Build Log: make-02.09.2022-01.46.log
Server Log: logs-02.09.2022-01.46.tar.gz

Finished