Building
Started building at 2022/09/02 01:46:26
Using pegged server, 1948 build
Calculating base
Updating mirror
Basing run on 7.2.0-1948 3df08568d
Updating tree for run 02.09.2022-01.46
query is at c3f3821, changes since last good build:
c3f3821ea MB-53533 Ensure initialisation is complete ahead of PrepareTopologyChange processing.
e74608abe MB-53541 Independent exchange notifier for merge actions.
6a1b677ce MB-53565 Do not consider primary index for index scan inside correlated subquery
89f2c4277 MB-53565 Proper formalization for ExpressionTerm and SubqueryTerm under ANSI join
3c0a17984 MB-53565 Do not use join predicates as index span if under hash join
f636e8c63 MB-53506 Additional fixes for unit tests
b5c561c25 MB-53506 Fix some unit test issues
6de332853 MB-53230. Change load_factor to moving avg
8fcba5ab8 MB-53410 Revise fix.
5ce5fd088 MB-53528 add SkipMetering to index connection
8047faf2e MB-53526 register unbillable admin units with regulator
cae373174 MB-53410 Reduce redundant value creation in UNNEST.
be0672272 MB-52176:If use_cbo setting is true then run update statistics after CREATE INDEX for non deferred indexes and after BUILD INDEX for deferred indexes
d0e4f965a updating go mod for building in module aware mode
f3f2f3580 MB-53514 Revise fix.
08344c66a MB-53514 Report KV RUs incurred for sequential scans
120933d52 Revert "MB-53514 Report KV RUs incurred for sequential scans"
f70f23c4f MB-53514 Report KV RUs incurred for sequential scans
63fbf9734 MB-53406 Reduce atomics in lockless pool.
c68ea6550 MB-53506 Prevent multiple mutations on a single key in UPSERT.
407a2027f MB-53477 Alter threshold for spilling group-by without a quota.
b7552a36e MB-53472:Add command line option -query_context or -qc to specify the query_context when connecting to the couchbase cluster to launch cbq shell
1613c282d MB-53506 Improve sequential scan integration with IUD operations.
c754bf3c9 MB-53176:Perform memory accounting when cached values are freed in the ExpressionScan operator
e178f11e6 MB-33748 Go fmt
174adf156 MB-33748 Making time in catalogs ISO-8601 format
73e10feeb MB-53230. Use minimum value
1c61edc90 MB-51928 Additional fixes for join filter
839c44620 MB-44219 credentials not always available to planner
1c9c1c6a1 MB-53439 MB-50997 scope UDF definitions stored in _system scope, open system:functions
456304363 MB-53446 Honor USE NL hint for kv range scan
d1fc29ec6 MB-53444 Avoid invalid numbers when calculating group cost
f2d314563 MB-44219: In serverless mode, check if user has permissions on bucket when executing a scoped function
b72782619 MB-53434. return false only on error
519566933 go mod tidy (update to bleve@v2.3.4)
c327709fe MB-52098: go mod tidy
ab8a702f2 MB-53427 cbq vim-mode: enhance commands
a85b778e2 MB-53420 consider empty non nil parameter list for inline UDF caching
568270f72 MB-53416. Set Conditional for NVL/NVL2/Decode/IfMissing/IfNull
6502af299 MB-53391 Fix value tracking for RAW projections
d1e41aa49 MB-29604 Fix test cases.
0b269607c MB-44305 Spill sort and group by to disk.
e8b2e1f49 MB-53394 Don't escape HTML in errors and warnings.
8d2e43a17 MB-29604 Return warning on division by zero.
6a6d16cb5 MB-53372 properly formalize select in UDF with variables as non correlated, avoid caching for UDF select
0faaadf0c MB-53377 Proper set up of join keys when converting from ANSI join to lookup join
f47e3d29b MB-53371 Use 64-bit value for 'Uses'
86c6a347a MB-53353 Adjust unbounded folllowing window range
039b1d27c GOCBC-1325 Add Transaction logger
c4f13bfb9 MB-52952 - [FTS] Improving the precision of stats used for WUs metering
4d2617f4a MB-44219 In serverless mode if the user is not an admin user and tries to access a collection, scope or bucket that does not exist or they do not have permissions on: If the user has permissions on the bucket in the keyspace they are trying to access, display the specific error message. If not, display the generic access denied message.
758bc9411 MB-52176 Perform automatic UPDATE STATISTICS for CREATE PRIMARY INDEX
38fa22e85 MB-53254 Ignore transaction binary documents in sequential scans
5dd58d359 MB-53291 Administrator is throttled
5fcf49bd5 MB-53298 Do not return error for MISSING value in join processing
8b1553f13 MB-53248 sub doc api to account for rus
34267f151 MB-52971 Handing CheckResultReject
0d0d028ec MB-53260 Fix last_scan_time for sequential scans.
70e3a059c -= passing TlsCAFile to regulator init (unused/deprecated)
c32b4fb19 "MB-53230 load factor of query node"
28049ab09 MB-51961 Fix typo
47cd8a06a Build failure Revert "MB-53230 load factor of query node"
46c894124 Revert "MB-44305 Spill sort and group by to disk."
9f11a3e8b MB-51961 Allow primary index to be used on inner of nested-loop join
95839d370 MB-44305 Spill sort and group by to disk.
4d1078dd1 MB-53230 load factor of query node
422c73fe7 MB-53235 leave item quota management to producer for serialized operators
gometa is at 05cb6b2, changes since last good build: none
ns_server is at 90035b0, changes since last good build:
90035b0e7 MB-52195 Tag "system" collections to not be metered
dd4f8b6de MB-52142: Fix default for bucket throttle setting
f5a520077 MB-50537: Don't allow strict n2n encryption if external unencrypted dist
823c23c4d MB-53478: Fix saving anonymous functions to disk
badb66c8f MB-52142: Add throttle limits to bucket config
f09932bb4 MB-52044 Fix eaccess crash
f2b6fed41 MB-52142: Add throttle limits to internal settings
2efa104ee MB-52350 Allow setting per-bucket storage limits
7e4c5811d Prevent memcached bucket creation on serverless
28db38600 MB-52350 Fix default values for storage limits
8fff21898 MB-51516: Don't clamp in_docs estimate to current checkpoint items
c2a03f4bf MB-52226: Introduce pause/resume APIs that are stubbed out
9cffcd45e MB-23768: Don't allow user backup in mixed clusters
f3e9ba2f0 MB-23768: Fix validator:json_array()
9e90c3142 MB-23768: Add menelaus_users:store_users and use it for restore
1f3a00032 MB-23768: Add replicated_dets:change_multiple/2
0c909be9c MB-23768: [rbac] Make sure we compress docs only once when ...
a804ff225 MB-23768: Add PUT /settings/rbac/backup
85a2b99a7 MB-23768: Move security_roles_access and ldap_access checks...
807549ea8 MB-23768: Fix validator:apply_multi_params
c85963d04 MB-23768: Add GET /settings/rbac/backup
253346b7b MB-23768: Call menelaus_roles:validate_roles when validating...
7b4f6c40e MB-23768: Remove unnecessary has_permission(Permission, Req) check
8301f1565 MB-23768: Remove permission param in verify_ldap_access
b08f974b7 MB-53326 Push CCCP payload on all kv nodes
19898a852 MB-52350 Fix unused variable
666af3431 MB-52350 Add storage limits to bucket config
8ae0cbb85 MB-52350 Add storage limits to internal settings
6b01ad610 MB-53423 Adjust bucket maximums for _system scope
44aa2ee1f MB-53288: New query node-quota parameter
207277058 MB-53352 Report the running config profile
65faa5fe7 MB-51738 Use this_node() in ns_memcached
3ab091f2e MB-51738 Define this_node() to handle distribution crash
f9693814a MB-53192: Add upgrade for memory alerts
763b16746 MB-53193: Reenable autofailover popup alerts
97fce2439 MB-47905: Pass client cert path to services
e9572c7c9 Update regulator frequently_changed_key prefix to /regulator/report
55725498f MB-53323: consider keep nodes when placing buckets in rebalance
492ae395e Add isServerless to /pools result
couchstore is at 803ec5f, changes since last good build:
803ec5f Refactor: mcbp::datatype moved to cb::mcbp::datatype
forestdb is at acba458, changes since last good build: none
kv_engine is at 1d85f5a, changes since last good build:
1d85f5a55 MB-53127: Document write should clear read usage
d69988fa9 MB-52311: [1/n] Pause / Resume Bucket: opcodes
225d4a7ea Refactor bucket delete to add extra extra unit tests
c14da3f90 MB-53510: Refactor bucket creation
8a3da42c2 MB-53543: Disable BackfillSmallBuffer test
cb7a5b432 MB-53304: Enforce holding of stateLock in VBucket::queueItem [2/3]
e74ba64e4 MB-53304: Enforce holding of stateLock in VBucket::queueItem [1/3]
0ceebade0 Remove ServerCallbackIface
19d61765d MB-52553: Don't special-case persistence cursor in CM::addStats
5836d9a70 MB-50984: Remove max_checkpoints hard limit on the single vbucket
47572e321 MB-50984: Default checkpoint_destruction_tasks=2
00201d310 MB-53523: Only check snap start vs last snap end if active VB
c90a937a4 Reformat test_reader_thread_starvation_warmup
dec1c6c2c Merge "Merge branch 'neo' into 'master'"
af74c95fe Refactor: CheckpointManager::registerCursorBySeqno()
a933cf568 MB-53448: DCP_ADD_STREAM_FLAG_TO_LATEST should use highSeqno of collections(s) in filter
08438bfb8 MB-53259: Update DCP Consumer byffer-size at dynamic Bucket Quota change
e61d09645 [Refactor] deselect bucket before trying to delete
afac71aab MB-53055: Fix Checkpoint::isEmptyByExpel() semantic
ae8baf2dc Remove unused code from kvstore_test
6f5ba689c [Refactor] Move bufferevent related code to subclass
2dd1745c6 MB-53498: Delay bucket type update
d50b99685 Merge branch 'neo' into 'master'
79de292f2 Merge branch 'neo' into 'master'
34bc1c7d2 Merge "Merge branch 'neo' into 'master'"
074db327f Enable KVStoreTest GetBySeqno for non-couchstore
16bd96ae6 MB-53284: Use magma memory optimized writes in BucketQuotaChangeTest
854eced08 Merge branch 'neo' into 'master'
50f5747b7 Merge "Merge branch 'neo' into 'master'"
c231af910 Cleanup: remove 'polling' durability timeout mode
f5930b3ea Tidy: Checkpoint::queueDirty use structured binding in for loop
a65ca2ba1 Merge branch 'neo' into 'master'
16f186be2 Only regenerate serverless/configuration.json if exe changed
194900077 MB-53052: Remove KVBucket::itemFreqDecayerIsSnoozed()
4b4ad639d Refactor: Create factory method for Connection objects
f3ac46848 MB-35297: Fix RangeScan sampling stats NotFound path
7d3f297f7 MB-46738: Rename dcp_conn_buffer_ratio into dcp_consumer_buffer_ratio
9bc866891 [Refactor] Remove the history field of sloppy gauge
a7b78c756 MB-53055: Add highestExpelledSeqno to Checkpoint ostream
a58bd636c MB-53055: Add highest_expelled_seqno to Checkpoint stats
7e4587d1e Remove duplicate method in DurabilityEPBucketTest
12136509b Add labels to Montonic<> members of Checkpoint
795dd8dc0 MB-53055: Fix exception message in CM::registerCursorBySeqno
200aa87ae Add "filter" capabilities to delete bucket
6fcfed646 SetClusterConfig should create config-only bucket
6990718c8 MB-52953: Remove refs to old replication-throttle params and stats
0fdcf8882 MB-52953: Remove unused EPStats::replicationThrottleThreshold
bc4592d8b MB-52953: Use mutation_mem_threshold in ReplicationThrottleEP::hasSomeMemory
b1ed0feb2 MB-52953: Turn mutation_mem_threshold into mutation_mem_ratio
45dd2db60 MB-53429: Hold vbState lock during pageOut
ba18b10ca MB-53438: Acquire the vbState lock during disk backfill
348287953 MB-53141: Return all if sampling range-scan requests samples > keys
9da38ff86 MB-35297: Improve logging for RangeScan create/cancel
79aa3dd72 MB-53100: Add extra seqno log information after we register a cursor
415b3ec74 MB-53198: Do not abort warmup for shard if scan cancelled
dc09bb535 Cleanup: Move mcbp::datatype to cb::mcbp::datatype
a77fca118 MB-35297: Meter RangeScan create
36d090abe MB-35297: Throttle RangeScan create/continue
40321cf27 SetClusterConfig should handle all bucket states
ac0c0486d Merge commit 'couchbase/neo~7' into trunk
c615d15f2 Merge "Merge commit 'couchbase/neo~10' into trunk"
a96e4a5e9 MB-52806: Disconnect DCP connections when they loose privilege
6b7d68b4e MB-52158: Check for privilege in RangeScan continue/cancel
b887f1f17 Merge commit 'couchbase/neo~10' into trunk
2cfe963a7 Modernize config parsing [2/2]
4a6018627 MB-53359: Add uniqe error code for config-bucket
e0e5d5c98 MB-35297: Add EventDrivenTimeoutTask
8bfdba483 Cleanup: move mcbp::subdoc under cb::mcbp::subdoc
cf97e6792 Cleanup: Move mcbp::cas under cb::mcbp::cas
3834eb115 MB-43127: Log succcess status from dumpCallback
bcb730456 MB-52172 Refactor source file generation cmake target
d847f8a55 MB-35297: Meter RangeScan key/values
a7a610b48 Refactor: Rename CreateBucketCommandContext
af47290a6 Refactor out wait code to separate method
3c30a1142 Include all bucket states in "bucket_details "
5d272f547 MB-53379: Allow Collection enabled clients to select COB
0042495b9 MB-52975: Fold backfill create and scan into one invocation of run
53f915d1d MB-35297: runtime must not be zero when backfill completes
a811f317b MB-53359: Don't try to fetch bucket metrics from config-only bucket
881774c5e MB-53354: Extend CheckpointMemoryTrackingTest suite for non-SSO case
7d7389df7 Modernize parse_config [1/2]
72e650860 Set the correct hostname for dcp metering test
8325ff14b Remove support for DT_CONFIGFILE
92c8f4fa8 Remove config_parse from server-api
f85f41bad MB-35297: RangeScan document 'flags' should match GetMeta byte order
3eccd2aa6 MB-53157: RangeScanCreate uuid should be a string
67d4759c0 MB-52953: Add ReplicationThrottleEP::engine member
c310b2f4a Don't use the term whitelist
407905037 MB-53197: Add support for ClusterConfigOnly bucket
f61b2e1c6 MB-53294: Introduce storage_bytes metering metric
be1577087 MB-52953: Remove unused UseActiveVBMemThreshold
ecbd40992 MB-35297: Add missing recvResponse / sendCommand from RangeScanTest/CreateInvalid
e3bbe2ace MB-52953: Use only mutation_mem_threshold in VB::hasMemoryForStoredValue
533286852 MB-53294: Refactor engine Prometheus metrics
03056b2d2 MB-53294: Rename Cardinality -> MetricGroup
8937d6e5a MB-52953: Default replication_throttle_threshold=93
6579346af MB-52956: Update lastReadSeqno at the end of an OSO backfill
3af167ac7 MB-52953: Move VBucket::mutationMemThreshold to KVBucket
8c5af9915 MB-52854: Fix and re-enable the DcpConsumerBufferAckTest suite
100a5b2af MB-52957: Avoid scan when collection high seqno < start
cd6df9b81 Make wasFirst in ActiveStream snapshot functions const
7bc7ee427 Sanity check that snap start > previous snap end
8f324c470 MB-53184: Extend range-scan computed exclusive-end upto the input
3f6fb6ba2 MB-46738: Remove Vbid arg from the buffer-ack DCP api
cdc3c2f29 MB-52842: Fix intermittent failure in 'disk>RAM delete paged-out'
1588cb007 Merge "Merge branch 'neo' into 'master'"
552d9e2c7 MB-46738: Remove unused dcp_conn_buffer_size_max
e44ee005e MB-46738: Remove unused dcp_conn_buffer_size
769d20940 MB-52264: Add desiredMaxSize stat
6809d7eae MB-46738: Ensure Consumer buffer size always ratio of bucket quota
b05ebef25 Merge branch 'neo' into 'master'
ab1ab27f8 Merge "Merge commit 'ea65052e' into 'couchbase/master'"
a6e70fdae Merge commit 'ea65052e' into 'couchbase/master'
503ae084b MB-46738: Make DcpFlowControlManager::engine const
e5766a51e MB-46738: Make dcp_conn_buffer_ratio dynamic
979159649 MB-53205: Hold VBucket stateLock while calling fetchValidValue
89602bce3 Humpty-Dumpty: Failover exploration tool
bb17d9439 MB-53197: [Refactor] create BucketManager::setClusterConfig
a81e37998 Upgrade go version to 1.19 for tls_test
256c78709 Merge "MB-52383: Merge branch 'cheshire-cat' into neo" into neo
09bbfce5c Merge "MB-47851: Merge branch 'cheshire-cat' into neo" into neo
e99ce1c4a Merge "MB-47267: Merge branch 'cheshire-cat' into neo" into neo
112e09c36 Merge "MB-51373: Merge branch 'cheshire-cat' into neo" into neo
281df3be1 Merge "Merge branch 'cheshire-cat' into neo" into neo
46014c72f MB-52383: Merge branch 'cheshire-cat' into neo
ecc2f6bb7 Change the logic for Unmetered privilege
c73eaf5f5 MB-53100: Add streamName arg to MockActiveStream ctor
ba7850f07 MB-47851: Merge branch 'cheshire-cat' into neo
5edb02327 MB-47267: Merge branch 'cheshire-cat' into neo
8db209a68 MB-51373: Merge branch 'cheshire-cat' into neo
eb865cbb0 Merge branch 'cheshire-cat' into neo
f656b5152 Merge "Merge branch 'cheshire-cat' into neo" into neo
453eb9a98 MB-53282: Reset open_time in early return in close_and_rotate_file
2a83a2a63 MB-52383: Merge branch 'mad-hatter' into cheshire-cat
9c684fb52 Merge branch 'mad-hatter' into cheshire-cat
852883091 Merge branch 'mad-hatter' into cheshire-cat
0173173cb Revert "MB-52813: Don't call Seek for every call of ::scan"
f1c3ddc67 Merge branch 'cheshire-cat' into neo
349c2640c Set GOVERSION to 1.18 to remove warning from cmake
abfb02f80 MB-46738: FCManager API takes DcpConsumer&
4ab7dbaa3 MB-52264: Wait for memory to reduce before setting new quota
d5d7b65d0 [serverless] Split Get metering test to individual tests
fda7ec6b8 Remove old comment in PagingVisitor
5d9bdbb44 MB-52633: Swap PagingVisitor freq counter histogram to flat array
f494fa983 MB-51373: Merge branch 'mad-hatter' into cheshire-cat
6d32e009a MB-52669: Specify GOVERSION without patch revision
df808528f Merge "Merge branch 'neo'"
eeb5cbad7 Merge branch 'neo'
18a4cd691 MB-52793: Merge branch 'mad-hatter' into cheshire-cat
b4c2fe22b Merge branch 'mad-hatter' into cheshire-cat
c80c6f58c MB-51373: Inspect and correct Item objects created by KVStore
ea65052eb MB-53046: [BP] Timeout SeqnoPersistenceRequests when no data is flushed
5f6d5dc65 MB-47267 / MB-52383: Make backfill during warmup a PauseResume task
4e51c38a8 MB-47851: Cancel any requests blocked on warmup if warmup stopped.
2c6e95c8e MB-47267: Make ObjectRegistry getAllocSize atomic
3d73de526 MB-52902: Populate kvstore rev if no vbstate found
ad47f53b7 MB-51373: Inspect and correct Item objects created by KVStore
8855aebe5 MB-52793: Ensure StoredValue::del updates datatype
35086bc80 Merge remote-tracking branch 'couchbase/alice' into mad-hatter
0df2087be MB-43055: [BP] Ensure ItemPager available is not left set to false
6dfd920a8 MB-43453: mcctl: Use passwd from env or stdin
b7d5bd362 MB-40531: [BP] Prefer paging from replicas if possible
Switching indexing to unstable
indexing is at 19ea81d, changes since last good build: none
Switching plasma to unstable
plasma is at cfa6534, changes since last good build:
fatal: Invalid revision range 0141641db3ee3de853547c46ed58c647fc7c43a1..HEAD
Switching nitro to unstable
nitro is at 966c610, changes since last good build: none
Switching gometa to master
gometa is at 05cb6b2, changes since last good build: none
Switching testrunner to master
Submodule 'gauntlet' (https://github.com/pavithra-mahamani/gauntlet) registered for path 'gauntlet'
Submodule 'java_sdk_client' (https://github.com/couchbaselabs/java_sdk_client) registered for path 'java_sdk_client'
Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI'
Submodule path 'gauntlet': checked out '4e2424851a59c6f4b4edfdb7e36fa6a0874d6300'
Submodule path 'java_sdk_client': checked out '961d8eb79ec29bad962b87425eca59fc43c6fe07'
Submodule path 'lib/capellaAPI': checked out '879091aa331e3d72f913b8192f563715d9e8597a'
testrunner is at f2361d1, changes since last good build: none
Pulling in uncommitted change 176468 at refs/changes/68/176468/14
Total 52 (delta 40), reused 47 (delta 40)
[unstable 5aed2c1a] MB-51825: Add NumVBuckets in TsVbuuid struct and fix tsvbuuid pool
Author: Sai Krishna Teja Kommaraju
Date: Tue Jun 21 22:27:17 2022 +0530
1 file changed, 44 insertions(+), 9 deletions(-)
Pulling in uncommitted change 176523 at refs/changes/23/176523/3
Total 51 (delta 40), reused 48 (delta 40)
[unstable 9b1d2106] MB-51825: Fix memedb_slice_test adding numVBuckets arg
Author: Sai Krishna Teja Kommaraju
Date: Wed Jun 22 16:31:49 2022 +0530
1 file changed, 1 insertion(+), 1 deletion(-)
Pulling in uncommitted change 178917 at refs/changes/17/178917/8
[unstable b8136e51] MB-51825: Pass numVBuckets to Storage from indexer
Author: Sai Krishna Teja Kommaraju
Date: Wed Aug 17 15:08:14 2022 +0530
5 files changed, 56 insertions(+), 26 deletions(-)
Pulling in uncommitted change 178948 at refs/changes/48/178948/10
[unstable 7971e62b] MB-51825: Indexer - Fetch numVBuckets from cinfo
Author: Sai Krishna Teja Kommaraju
Date: Wed Aug 17 20:53:00 2022 +0530
2 files changed, 57 insertions(+), 21 deletions(-)
Pulling in uncommitted change 178949 at refs/changes/49/178949/10
[unstable 1e0c2545] MB-51825: Scan_Coordinator - Fetch numVbuckets from cinfo
Author: Sai Krishna Teja Kommaraju
Date: Wed Aug 17 22:26:59 2022 +0530
2 files changed, 14 insertions(+), 11 deletions(-)
Pulling in uncommitted change 178950 at refs/changes/50/178950/11
[unstable 577e8f67] MB-51825: Storage_Manager - Fetch numVbuckets from cinfo
Author: Sai Krishna Teja Kommaraju
Date: Wed Aug 17 23:33:14 2022 +0530
6 files changed, 102 insertions(+), 17 deletions(-)
Pulling in uncommitted change 178951 at refs/changes/51/178951/11
[unstable 3fe0958b] MB-51825: Mutation_Manager - Fetch numVbuckets from cinfo
Author: Sai Krishna Teja Kommaraju
Date: Thu Aug 18 00:06:03 2022 +0530
4 files changed, 37 insertions(+), 22 deletions(-)
Pulling in uncommitted change 178958 at refs/changes/58/178958/11
[unstable 72e32fb5] MB-51825: TimeKeeper - Fetch numVbuckets from cinfo
Author: Sai Krishna Teja Kommaraju
Date: Thu Aug 18 04:46:09 2022 +0530
3 files changed, 59 insertions(+), 44 deletions(-)
Pulling in uncommitted change 179474 at refs/changes/74/179474/1
Total 74 (delta 60), reused 70 (delta 60)
[unstable c074bb7d] MB100 : Add function to satisfy datastore.Context interface
Author: Sai Krishna Teja Kommaraju
Date: Thu Sep 1 22:25:22 2022 +0530
4 files changed, 16 insertions(+)
Pulling in uncommitted change 178623 at refs/changes/23/178623/14
Total 6 (delta 1), reused 2 (delta 1)
[unstable eec4521] MB-52020: Add Shard Locking APIs
Author: saptarshi.sen
Date: Fri Aug 5 11:27:27 2022 -0700
4 files changed, 755 insertions(+), 29 deletions(-)
Pulling in uncommitted change 179000 at refs/changes/00/179000/9
Total 17 (delta 10), reused 11 (delta 10)
[unstable 9739a4c] MB-52020: Add Shard CopyStats
Author: saptarshi.sen
Date: Wed Aug 10 15:03:32 2022 -0700
9 files changed, 531 insertions(+), 114 deletions(-)
Pulling in uncommitted change 179007 at refs/changes/07/179007/7
Total 24 (delta 17), reused 19 (delta 17)
[unstable 59d382d] MB-52020: Add TransferShard & RestoreShard API
Author: saptarshi.sen
Date: Fri Aug 19 00:30:05 2022 -0700
5 files changed, 1884 insertions(+), 80 deletions(-)
Pulling in uncommitted change 179316 at refs/changes/16/179316/4
Total 32 (delta 23), reused 27 (delta 23)
[unstable c7eb7d4] MB-52020: Handle Instance Paths in Shard Metadata Snapshot
Author: saptarshi.sen
Date: Sat Aug 27 22:23:31 2022 -0700
6 files changed, 772 insertions(+), 278 deletions(-)
Pulling in uncommitted change 179442 at refs/changes/42/179442/2
Total 38 (delta 29), reused 34 (delta 29)
[unstable a15a1b0] MB-52020: Handle zero segmentSize in persistent config snapshot
Author: saptarshi.sen
Date: Wed Aug 31 11:19:16 2022 -0700
4 files changed, 13 insertions(+), 16 deletions(-)
Pulling in uncommitted change 179443 at refs/changes/43/179443/2
Total 44 (delta 36), reused 42 (delta 36)
[unstable fee3f71] MB-52020: Add checksum to persistent config snapshot
Author: saptarshi.sen
Date: Wed Aug 31 07:50:00 2022 -0700
4 files changed, 114 insertions(+), 15 deletions(-)
Building community edition
Building cmakefiles and deps [CE]
Building main product [CE]
Build CE finished
BUILD_ENTERPRISE empty. Building enterprise edition
Building Enterprise Edition
Building cmakefiles and deps [EE]
Building main product [EE]
Build EE finished
Testing
Started testing at 2022/09/02 02:35:34
Testing mode: sanity,unit,functional,integration
Using storage type: memdb
Setting ulimit to 200000
Simple Test
Sep 02 02:40:35 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 02 02:44:26 rebalance_in_with_ops (rebalance.rebalancein.RebalanceInTests) ... ok
Sep 02 02:45:11 do_warmup_100k (memcapable.WarmUpMemcachedTest) ... ok
Sep 02 02:46:41 test_view_ops (view.createdeleteview.CreateDeleteViewTests) ... ok
Sep 02 02:47:34 b" 'stop_on_failure': 'True'}"
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 02 02:47:34 b"{'nodes_in': '3', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_1'}"
Sep 02 02:47:34 b'-->result: '
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 1 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=3,bucket_type=ephemeral,replicas=1,items=50000,get-logs-cluster-run=True,doc_ops=create;update;delete'
Sep 02 02:47:34 b"{'nodes_in': '3', 'bucket_type': 'ephemeral', 'replicas': '1', 'items': '50000', 'get-logs-cluster-run': 'True', 'doc_ops': 'create;update;delete', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_2'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t memcapable.WarmUpMemcachedTest.do_warmup_100k,get-logs-cluster-run=True'
Sep 02 02:47:34 b"{'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_3'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 02:47:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.createdeleteview.CreateDeleteViewTests.test_view_ops,ddoc_ops=create,test_with_view=True,num_ddocs=1,num_views_per_ddoc=10,items=1000,skip_cleanup=False,get-logs-cluster-run=True'
Sep 02 02:47:34 b"{'ddoc_ops': 'create', 'test_with_view': 'True', 'num_ddocs': '1', 'num_views_per_ddoc': '10', 'items': '1000', 'skip_cleanup': 'False', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_4'}"
Sep 02 02:47:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 02:47:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 02:47:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 02:57:31 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1,timeout=1200,get-logs-cluster-run=True'ok
Sep 02 02:58:13 test_simple_dataset_stale_queries_data_modification (view.viewquerytests.ViewQueryTests) ... ok
Sep 02 03:01:58 load_with_ops (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:05:52 load_with_failover (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:08:34 suite_tearDown (xdcr.uniXDCR.unidirectional) ... ok
Sep 02 03:08:34 b"{'num_nodes_to_add': '1', 'skip_rebalance': 'true', 'docs-per-day': '1', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_5'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 1 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification,num-docs=1000,skip_rebalance=true,timeout=1200,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'num-docs': '1000', 'skip_rebalance': 'true', 'timeout': '1200', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_6'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'replicas': '1', 'items': '10000', 'value_size': '128', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_7'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 1 , fail 0'
Sep 02 03:08:34 b'./testrunner -i b/resources/dev-4-nodes-xdcr.ini -p makefile=True,stop_on_failure=True,log_level=CRITICAL -t xdcr.uniXDCR.unidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=unidirection,doc-ops=update-delete,failover=source,get-logs-cluster-run=True'
Sep 02 03:08:34 b"{'replicas': '1', 'items': '10000', 'ctopology': 'chain', 'rdirection': 'unidirection', 'doc-ops': 'update-delete', 'failover': 'source', 'get-logs-cluster-run': 'True', 'ini': 'b/resources/dev-4-nodes-xdcr.ini', 'cluster_name': 'dev-4-nodes-xdcr', 'spec': 'simple', 'conf_file': 'conf/simple.conf', 'makefile': 'True', 'stop_on_failure': 'True', 'log_level': 'CRITICAL', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 8, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_02-35-55/test_8'}"
Sep 02 03:08:34 b'summary so far suite rebalance.rebalancein.RebalanceInTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite memcapable.WarmUpMemcachedTest , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.createdeleteview.CreateDeleteViewTests , pass 1 , fail 0'
Sep 02 03:08:34 b'summary so far suite view.viewquerytests.ViewQueryTests , pass 2 , fail 0'
Sep 02 03:08:34 b'summary so far suite xdcr.uniXDCR.unidirectional , pass 2 , fail 0'
Sep 02 03:08:34 b'Run after suite setup for xdcr.uniXDCR.unidirectional.load_with_failover'
Sep 02 03:08:34 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 02 03:08:34 b"('rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops', ' pass')"
Sep 02 03:08:34 b"('memcapable.WarmUpMemcachedTest.do_warmup_100k', ' pass')"
Sep 02 03:08:34 b"('view.createdeleteview.CreateDeleteViewTests.test_view_ops', ' pass')"
Sep 02 03:08:34 b"('view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in', ' pass')"
Sep 02 03:08:34 b"('view.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries_data_modification', ' pass')"
Sep 02 03:08:34 b"('xdcr.uniXDCR.unidirectional.load_with_ops', ' pass')"
Sep 02 03:08:34 b"('xdcr.uniXDCR.unidirectional.load_with_failover', ' pass')"
Unit tests
=== RUN TestMerger
--- PASS: TestMerger (0.01s)
=== RUN TestInsert
--- PASS: TestInsert (0.00s)
=== RUN TestInsertPerf
16000000 items took 29.763671549s -> 537568.0877830937 items/s conflicts 23
--- PASS: TestInsertPerf (29.76s)
=== RUN TestGetPerf
16000000 items took 5.401702236s -> 2.9620292457749606e+06 items/s
--- PASS: TestGetPerf (5.99s)
=== RUN TestGetRangeSplitItems
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.3351,
"memory_used": 37360992,
"node_allocs": 1000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 749144,
"level1": 187776,
"level2": 47222,
"level3": 11903,
"level4": 2974,
"level5": 735,
"level6": 182,
"level7": 48,
"level8": 12,
"level9": 2,
"level10": 2,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Split range keys [35911 108875 357987 462344 598654 671258 781160]
No of items in each range [35911 72964 249112 104357 136310 72604 109902 218840]
--- PASS: TestGetRangeSplitItems (0.59s)
=== RUN TestBuilder
{
"node_count": 50000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.3333,
"memory_used": 1866650928,
"node_allocs": 50000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 37500395,
"level1": 9374070,
"level2": 2344403,
"level3": 586130,
"level4": 146617,
"level5": 36338,
"level6": 9033,
"level7": 2299,
"level8": 539,
"level9": 123,
"level10": 39,
"level11": 9,
"level12": 5,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 9.954860057s to build 50000000 items, 5.0226725e+06 items/sec
Took 693.201997ms to iterate 50000000 items
--- PASS: TestBuilder (10.65s)
=== RUN TestNodeDCAS
--- PASS: TestNodeDCAS (0.00s)
PASS
ok github.com/couchbase/indexing/secondary/memdb/skiplist 47.253s
=== RUN TestInsert
Set IO Concurrency: 716
--- PASS: TestInsert (0.01s)
=== RUN TestInsertPerf
20000000 items took 49.431309719s -> 404601.862942599 items/s snapshots_created 210 live_snapshots 1
--- PASS: TestInsertPerf (62.00s)
=== RUN TestInsertDuplicates
--- PASS: TestInsertDuplicates (0.01s)
=== RUN TestGetPerf
16000000 items took 11.188772578s -> 1.4300049347200163e+06 items/s
--- PASS: TestGetPerf (12.82s)
=== RUN TestLoadStoreDisk
Inserting 1000000 items took 1.513861451s
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 8,
"next_pointers_per_node": 1.3321,
"memory_used": 57313740,
"node_allocs": 999999,
"node_frees": 0,
"level_node_distribution":{
"level0": 750475,
"level1": 187509,
"level2": 46677,
"level3": 11462,
"level4": 2907,
"level5": 715,
"level6": 190,
"level7": 48,
"level8": 15,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
2022-09-02T03:10:51.986+05:30 [Info] MemDB::StoreToDisk: Unexpected amount of data being persisted dir [db.dump] count per shard [[39804 88146 47888 80654 56585 14456 188193 207080 242220 34974 0 0 0 0 0 0]] total count [1000000] snap count [999999] size per shard [[318432 705168 383104 645232 452680 115648 1505544 1656640 1937760 279792 0 0 0 0 0 0]] total size [8000000] memoryInUse [57313740]
2022-09-02T03:10:52.058+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[39804 88146 47888 80654 56585 14456 188193 207080 242220 34974 0 0 0 0 0 0]] total count [1000000] snap count [999999] size per shard [[318432 705168 383104 645232 452680 115648 1505544 1656640 1937760 279792 0 0 0 0 0 0]] total size [8000000] memoryInUse [57313740]
Storing to disk took 330.092005ms
Loading from disk took 291.417456ms
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.3330,
"memory_used": 57328656,
"node_allocs": 1000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 750089,
"level1": 187546,
"level2": 46769,
"level3": 11662,
"level4": 2997,
"level5": 703,
"level6": 184,
"level7": 38,
"level8": 10,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestLoadStoreDisk (3.44s)
=== RUN TestConcurrentLoadCloseFragmentation
Done Inserting 2000000 items
frag = NaN%
2022-09-02T03:10:57.465+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[2672 151944 85242 108240 50825 171243 119129 99979 175995 314459 55722 36183 14087 102727 70780 440773]] total count [2000000] snap count [2000000] size per shard [[21376 1215552 681936 865920 406600 1369944 953032 799832 1407960 2515672 445776 289464 112696 821816 566240 3526184]] total size [16000000] memoryInUse [114671996]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentation (5.53s)
=== RUN TestConcurrentLoadCloseFragmentationSmall
Done Inserting 10 items
frag = NaN%
2022-09-02T03:10:59.100+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationSmall (0.09s)
=== RUN TestConcurrentLoadCloseFragmentationEmpty
Done Inserting 0 items
frag = NaN%
2022-09-02T03:10:59.197+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total count [0] snap count [0] size per shard [[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]] total size [0] memoryInUse [0]
Done Storing to disk
frag = NaN%
Done Closing
Done Loading from disk
frag = NaN%
--- PASS: TestConcurrentLoadCloseFragmentationEmpty (0.06s)
=== RUN TestConcurrentCloseSingleNode
A
--- PASS: TestConcurrentCloseSingleNode (0.00s)
=== RUN TestStoreDiskShutdown
Inserting 1000000 items took 1.591866603s
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 19,
"next_pointers_per_node": 1.3337,
"memory_used": 57339584,
"node_allocs": 1000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 749769,
"level1": 187619,
"level2": 46921,
"level3": 11775,
"level4": 2953,
"level5": 722,
"level6": 182,
"level7": 49,
"level8": 9,
"level9": 1,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
snap count = 1000000
Prepare returns error: MemDB instance has been shutdown
--- PASS: TestStoreDiskShutdown (2.36s)
=== RUN TestDelete
{
"node_count": 10,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.2000,
"memory_used": 572,
"node_allocs": 10,
"node_frees": 0,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
{
"node_count": 10,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.2000,
"memory_used": 572,
"node_allocs": 20,
"node_frees": 10,
"level_node_distribution":{
"level0": 8,
"level1": 2,
"level2": 0,
"level3": 0,
"level4": 0,
"level5": 0,
"level6": 0,
"level7": 0,
"level8": 0,
"level9": 0,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
--- PASS: TestDelete (1.00s)
=== RUN TestGCPerf
final_node_count = 16000, average_live_node_count = 68342, wait_time_for_collection = 20ms
--- PASS: TestGCPerf (36.64s)
=== RUN TestMemoryInUse
ItemsCount: 5000, MemoryInUse: 297328, NodesCount: 5000
ItemsCount: 0, MemoryInUse: 64, NodesCount: 0
--- PASS: TestMemoryInUse (1.01s)
=== RUN TestFullScan
Inserting 1600000 items took 2.638705882s
{
"node_count": 1600000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 6,
"next_pointers_per_node": 1.3336,
"memory_used": 91740400,
"node_allocs": 1600000,
"node_frees": 0,
"level_node_distribution":{
"level0": 1199521,
"level1": 300382,
"level2": 75072,
"level3": 18867,
"level4": 4639,
"level5": 1137,
"level6": 289,
"level7": 73,
"level8": 18,
"level9": 2,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Full iteration of 1600000 items took 257.556223ms
--- PASS: TestFullScan (4.06s)
=== RUN TestVisitor
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.3331,
"memory_used": 57329040,
"node_allocs": 1000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 750011,
"level1": 187841,
"level2": 46531,
"level3": 11594,
"level4": 3049,
"level5": 729,
"level6": 192,
"level7": 41,
"level8": 9,
"level9": 2,
"level10": 1,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Took 79.130347ms to iterate 1000000 items, 1.2637377e+07 items/s
shard - 0 count = 25442, range: 0-25441
shard - 1 count = 37610, range: 25442-63051
shard - 2 count = 21545, range: 63052-84596
shard - 3 count = 29031, range: 84597-113627
shard - 4 count = 18696, range: 113628-132323
shard - 5 count = 24128, range: 132324-156451
shard - 6 count = 105793, range: 156452-262244
shard - 7 count = 60804, range: 262245-323048
shard - 8 count = 17942, range: 323049-340990
shard - 9 count = 12013, range: 340991-353003
shard - 10 count = 62502, range: 353004-415505
shard - 11 count = 50301, range: 415506-465806
shard - 12 count = 26097, range: 465807-491903
shard - 13 count = 64785, range: 491904-556688
shard - 14 count = 42807, range: 556689-599495
shard - 15 count = 25019, range: 599496-624514
shard - 16 count = 13207, range: 624515-637721
shard - 17 count = 50415, range: 637722-688136
shard - 18 count = 3055, range: 688137-691191
shard - 19 count = 9842, range: 691192-701033
shard - 20 count = 15039, range: 701034-716072
shard - 21 count = 20247, range: 716073-736319
shard - 22 count = 27071, range: 736320-763390
shard - 23 count = 61815, range: 763391-825205
shard - 24 count = 66250, range: 825206-891455
shard - 25 count = 63032, range: 891456-954487
shard - 26 count = 19382, range: 954488-973869
shard - 27 count = 26130, range: 973870-999999
shard - 28 count = 0, range: 0-0
shard - 29 count = 0, range: 0-0
shard - 30 count = 0, range: 0-0
shard - 31 count = 0, range: 0-0
--- PASS: TestVisitor (1.53s)
=== RUN TestVisitorError
--- PASS: TestVisitorError (0.13s)
=== RUN TestLoadDeltaStoreDisk
snap count = 1000000
2022-09-02T03:11:50.171+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[0 0 0 0 0 104061 39234 180101 39811 110754 59093 119300 95316 120418 51620 59049]] total count [978757] snap count [1000000] size per shard [[0 0 0 0 0 832488 313872 1440808 318488 886032 472744 954400 762528 963344 412960 472392]] total size [7830056] memoryInUse [127390636]
Storing to disk took 15.393647843s
Loading from disk took 800.052422ms
{
"node_count": 1000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 0,
"next_pointers_per_node": 1.3346,
"memory_used": 57354336,
"node_allocs": 1000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 749046,
"level1": 188067,
"level2": 47299,
"level3": 11690,
"level4": 2920,
"level5": 723,
"level6": 196,
"level7": 42,
"level8": 11,
"level9": 3,
"level10": 2,
"level11": 1,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Restored 21243
RestoredFailed 463660
--- PASS: TestLoadDeltaStoreDisk (18.29s)
=== RUN TestExecuteConcurrGCWorkers
--- PASS: TestExecuteConcurrGCWorkers (0.65s)
=== RUN TestCloseWithActiveIterators
--- PASS: TestCloseWithActiveIterators (6.72s)
=== RUN TestDiskCorruption
Inserting 100000 items took 106.346529ms
{
"node_count": 100000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 10,
"next_pointers_per_node": 1.3338,
"memory_used": 5734012,
"node_allocs": 99998,
"node_frees": 0,
"level_node_distribution":{
"level0": 74940,
"level1": 18901,
"level2": 4548,
"level3": 1204,
"level4": 303,
"level5": 79,
"level6": 16,
"level7": 5,
"level8": 3,
"level9": 1,
"level10": 0,
"level11": 0,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
2022-09-02T03:12:11.788+05:30 [Info] MemDB::StoreToDisk: Done dir [db.dump] disk snapshot - count per shard [[9307 13012 820 3760 10901 18931 12803 5356 493 13371 426 3315 6682 823 0 0]] total count [100000] snap count [100000] size per shard [[74456 104096 6560 30080 87208 151448 102424 42848 3944 106968 3408 26520 53456 6584 0 0]] total size [800000] memoryInUse [5734012]
Storing to disk took 83.472155ms
Loading from disk took 357.50594ms
--- PASS: TestDiskCorruption (0.62s)
=== RUN TestSnapshotStats
--- PASS: TestSnapshotStats (0.00s)
=== RUN TestInsertDeleteConcurrent
total items in snapshot: 8393413 items deleted: 1606587
{
"node_count": 8393413,
"soft_deletes": 0,
"read_conflicts": 241,
"insert_conflicts": 2,
"next_pointers_per_node": 1.3334,
"memory_used": 481227412,
"node_allocs": 10000000,
"node_frees": 10000002,
"level_node_distribution":{
"level0": 6295269,
"level1": 1573247,
"level2": 393481,
"level3": 98677,
"level4": 24591,
"level5": 6094,
"level6": 1522,
"level7": 390,
"level8": 103,
"level9": 31,
"level10": 6,
"level11": 2,
"level12": 0,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
allocs: 20000002 frees: 20000002
--- PASS: TestInsertDeleteConcurrent (18.12s)
=== RUN TestNodeList
--- PASS: TestNodeList (0.00s)
PASS
ok github.com/couchbase/indexing/secondary/memdb 175.391s
=== RUN TestInteger
--- PASS: TestInteger (0.00s)
=== RUN TestSmallDecimal
--- PASS: TestSmallDecimal (0.00s)
=== RUN TestLargeDecimal
--- PASS: TestLargeDecimal (0.00s)
=== RUN TestFloat
--- PASS: TestFloat (0.00s)
=== RUN TestSuffixCoding
--- PASS: TestSuffixCoding (0.00s)
=== RUN TestCodecLength
--- PASS: TestCodecLength (0.00s)
=== RUN TestSpecialString
--- PASS: TestSpecialString (0.00s)
=== RUN TestCodecNoLength
--- PASS: TestCodecNoLength (0.00s)
=== RUN TestCodecJSON
--- PASS: TestCodecJSON (0.00s)
=== RUN TestReference
--- PASS: TestReference (0.00s)
=== RUN TestN1QLEncode
--- PASS: TestN1QLEncode (0.00s)
=== RUN TestArrayExplodeJoin
--- PASS: TestArrayExplodeJoin (0.00s)
=== RUN TestN1QLDecode
--- PASS: TestN1QLDecode (0.00s)
=== RUN TestN1QLDecode2
--- PASS: TestN1QLDecode2 (0.00s)
=== RUN TestArrayExplodeJoin2
--- PASS: TestArrayExplodeJoin2 (0.00s)
=== RUN TestMB28956
--- PASS: TestMB28956 (0.00s)
=== RUN TestFixEncodedInt
--- PASS: TestFixEncodedInt (0.00s)
=== RUN TestN1QLDecodeLargeInt64
--- PASS: TestN1QLDecodeLargeInt64 (0.00s)
=== RUN TestMixedModeFixEncodedInt
TESTING [4111686018427387900, -8223372036854775808, 822337203685477618]
PASS
TESTING [0]
PASS
TESTING [0.0]
PASS
TESTING [0.0000]
PASS
TESTING [0.0000000]
PASS
TESTING [-0]
PASS
TESTING [-0.0]
PASS
TESTING [-0.0000]
PASS
TESTING [-0.0000000]
PASS
TESTING [1]
PASS
TESTING [20]
PASS
TESTING [3456]
PASS
TESTING [7645000]
PASS
TESTING [9223372036854775807]
PASS
TESTING [9223372036854775806]
PASS
TESTING [9223372036854775808]
PASS
TESTING [92233720368547758071234000]
PASS
TESTING [92233720368547758071234987437653]
PASS
TESTING [12300000000000000000000000000000056]
PASS
TESTING [12300000000000000000000000000000000]
PASS
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056]
PASS
TESTING [123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]
PASS
TESTING [12300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056]
PASS
TESTING [210690]
PASS
TESTING [90000]
PASS
TESTING [123000000]
PASS
TESTING [3.60e2]
PASS
TESTING [36e2]
PASS
TESTING [1.9999999999e10]
PASS
TESTING [1.99999e10]
PASS
TESTING [1.99999e5]
PASS
TESTING [0.00000000000012e15]
PASS
TESTING [7.64507352e8]
PASS
TESTING [9.2233720368547758071234987437653e31]
PASS
TESTING [2650e-1]
PASS
TESTING [26500e-1]
PASS
TESTING [-1]
PASS
TESTING [-20]
PASS
TESTING [-3456]
PASS
TESTING [-7645000]
PASS
TESTING [-9223372036854775808]
PASS
TESTING [-9223372036854775807]
PASS
TESTING [-9223372036854775806]
PASS
TESTING [-9223372036854775809]
PASS
TESTING [-92233720368547758071234000]
PASS
TESTING [-92233720368547758071234987437653]
PASS
TESTING [-12300000000000000000000000000000056]
PASS
TESTING [-12300000000000000000000000000000000]
PASS
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000056]
PASS
TESTING [-123000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000]
PASS
TESTING [-210690]
PASS
TESTING [-90000]
PASS
TESTING [-123000000]
PASS
TESTING [-3.60e2]
PASS
TESTING [-36e2]
PASS
TESTING [-1.9999999999e10]
PASS
TESTING [-1.99999e10]
PASS
TESTING [-1.99999e5]
PASS
TESTING [-0.00000000000012e15]
PASS
TESTING [-2650e-1]
PASS
TESTING [-26500e-1]
PASS
TESTING [0.03]
PASS
TESTING [198.60]
PASS
TESTING [2000045.178]
PASS
TESTING [1.7976931348623157e+308]
PASS
TESTING [0.000000000000000000890]
PASS
TESTING [257953786.9864236576]
PASS
TESTING [257953786.9864236576e8]
PASS
TESTING [36.912e3]
PASS
TESTING [2761.67e0]
PASS
TESTING [2761.67e00]
PASS
TESTING [2761.67e000]
PASS
TESTING [7676546.67e-3]
PASS
TESTING [-0.03]
PASS
TESTING [-198.60]
PASS
TESTING [-2000045.178]
PASS
TESTING [-1.7976931348623157e+308]
PASS
TESTING [-0.000000000000000000890]
PASS
TESTING [-257953786.9864236576]
PASS
TESTING [-257953786.9864236576e8]
PASS
TESTING [-36.912e3]
PASS
TESTING [-2761.67e0]
PASS
TESTING [-2761.67e00]
PASS
TESTING [-2761.67e000]
PASS
TESTING [-7676546.67e-3]
PASS
--- PASS: TestMixedModeFixEncodedInt (0.01s)
=== RUN TestCodecDesc
--- PASS: TestCodecDesc (0.00s)
=== RUN TestCodecDescPropLen
--- PASS: TestCodecDescPropLen (0.00s)
=== RUN TestCodecDescSplChar
--- PASS: TestCodecDescSplChar (0.00s)
PASS
ok github.com/couchbase/indexing/secondary/collatejson 0.034s
Initializing write barrier = 8000
=== RUN TestForestDBIterator
2022-09-02T03:12:35.306+05:30 [INFO][FDB] Forestdb blockcache size 134217728 initialized in 5234 us
2022-09-02T03:12:35.307+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-02T03:12:35.310+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIterator (0.01s)
=== RUN TestForestDBIteratorSeek
2022-09-02T03:12:35.311+05:30 [INFO][FDB] Forestdb opened database file test
2022-09-02T03:12:35.314+05:30 [INFO][FDB] Forestdb closed database file test
--- PASS: TestForestDBIteratorSeek (0.00s)
=== RUN TestPrimaryIndexEntry
--- PASS: TestPrimaryIndexEntry (0.00s)
=== RUN TestSecondaryIndexEntry
--- PASS: TestSecondaryIndexEntry (0.00s)
=== RUN TestPrimaryIndexEntryMatch
--- PASS: TestPrimaryIndexEntryMatch (0.00s)
=== RUN TestSecondaryIndexEntryMatch
--- PASS: TestSecondaryIndexEntryMatch (0.00s)
=== RUN TestLongDocIdEntry
--- PASS: TestLongDocIdEntry (0.00s)
=== RUN TestMemDBInsertionPerf
Maximum number of file descriptors = 200000
Set IO Concurrency: 7200
Initial build: 10000000 items took 1m29.063007786s -> 112280.06159446055 items/s
Incr build: 10000000 items took 2m18.958197091s -> 71964.08854852419 items/s
Main Index: {
"node_count": 18000000,
"soft_deletes": 0,
"read_conflicts": 0,
"insert_conflicts": 7,
"next_pointers_per_node": 1.3333,
"memory_used": 1695887740,
"node_allocs": 18000000,
"node_frees": 0,
"level_node_distribution":{
"level0": 13500190,
"level1": 3375549,
"level2": 843074,
"level3": 210794,
"level4": 52734,
"level5": 13348,
"level6": 3245,
"level7": 782,
"level8": 221,
"level9": 42,
"level10": 17,
"level11": 3,
"level12": 1,
"level13": 0,
"level14": 0,
"level15": 0,
"level16": 0,
"level17": 0,
"level18": 0,
"level19": 0,
"level20": 0,
"level21": 0,
"level22": 0,
"level23": 0,
"level24": 0,
"level25": 0,
"level26": 0,
"level27": 0,
"level28": 0,
"level29": 0,
"level30": 0,
"level31": 0,
"level32": 0
}
}
Back Index 0 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 1 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 2 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 3 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 4 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 5 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 6 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 7 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 8 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 9 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 10 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 11 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 12 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 13 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 14 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
Back Index 15 : {
"FastHTCount": 625000,
"SlowHTCount": 0,
"Conflicts": 0,
"MemoryInUse": 26250000
}
--- PASS: TestMemDBInsertionPerf (228.02s)
=== RUN TestBasicsA
--- PASS: TestBasicsA (0.00s)
=== RUN TestSizeA
--- PASS: TestSizeA (0.00s)
=== RUN TestSizeWithFreelistA
--- PASS: TestSizeWithFreelistA (0.00s)
=== RUN TestDequeueUptoSeqnoA
--- PASS: TestDequeueUptoSeqnoA (0.10s)
=== RUN TestDequeueA
--- PASS: TestDequeueA (1.21s)
=== RUN TestMultipleVbucketsA
--- PASS: TestMultipleVbucketsA (0.00s)
=== RUN TestDequeueUptoFreelistA
--- PASS: TestDequeueUptoFreelistA (0.00s)
=== RUN TestDequeueUptoFreelistMultVbA
--- PASS: TestDequeueUptoFreelistMultVbA (0.00s)
=== RUN TestConcurrentEnqueueDequeueA
--- PASS: TestConcurrentEnqueueDequeueA (0.00s)
=== RUN TestConcurrentEnqueueDequeueA1
--- PASS: TestConcurrentEnqueueDequeueA1 (10.01s)
=== RUN TestEnqueueAppCh
--- PASS: TestEnqueueAppCh (2.00s)
=== RUN TestDequeueN
--- PASS: TestDequeueN (0.00s)
=== RUN TestConcurrentEnqueueDequeueN
--- PASS: TestConcurrentEnqueueDequeueN (0.00s)
=== RUN TestConcurrentEnqueueDequeueN1
--- PASS: TestConcurrentEnqueueDequeueN1 (10.01s)
PASS
ok github.com/couchbase/indexing/secondary/indexer 252.055s
=== RUN TestConnPoolBasicSanity
2022-09-02T03:16:50.502+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 3 overflow 6 low WM 3 relConn batch size 1 ...
2022-09-02T03:16:50.710+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:16:51.504+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnPoolBasicSanity (5.00s)
=== RUN TestConnRelease
2022-09-02T03:16:55.507+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Waiting for connections to get released
Waiting for more connections to get released
Waiting for further more connections to get released
2022-09-02T03:17:35.274+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:17:35.522+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestConnRelease (43.77s)
=== RUN TestLongevity
2022-09-02T03:17:39.276+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Releasing 1 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 4 conns.
Getting 1 conns.
Releasing 2 conns.
Getting 4 conns.
Releasing 3 conns.
Getting 4 conns.
Releasing 1 conns.
Getting 0 conns.
Releasing 2 conns.
Getting 1 conns.
Releasing 0 conns.
Getting 1 conns.
Releasing 3 conns.
Getting 3 conns.
Releasing 2 conns.
Getting 2 conns.
Releasing 2 conns.
Getting 3 conns.
Releasing 0 conns.
Getting 0 conns.
2022-09-02T03:18:17.674+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:18:18.292+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLongevity (42.40s)
=== RUN TestSustainedHighConns
2022-09-02T03:18:21.676+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 500 overflow 10 low WM 40 relConn batch size 10 ...
Allocating 16 Connections
cp.curActConns = 0
Returning 3 Connections
cp.curActConns = 11
Returning 2 Connections
cp.curActConns = 11
Allocating 6 Connections
Returning 4 Connections
cp.curActConns = 13
Returning 1 Connections
Allocating 12 Connections
cp.curActConns = 22
Returning 1 Connections
cp.curActConns = 23
Allocating 10 Connections
Returning 1 Connections
cp.curActConns = 32
Returning 3 Connections
Allocating 15 Connections
cp.curActConns = 33
Returning 4 Connections
cp.curActConns = 40
Returning 3 Connections
Allocating 8 Connections
cp.curActConns = 44
Returning 2 Connections
cp.curActConns = 43
Allocating 3 Connections
Returning 4 Connections
cp.curActConns = 42
Allocating 9 Connections
Returning 3 Connections
cp.curActConns = 48
Returning 2 Connections
Allocating 21 Connections
cp.curActConns = 55
Returning 4 Connections
cp.curActConns = 63
Returning 4 Connections
Allocating 0 Connections
cp.curActConns = 59
Returning 0 Connections
Allocating 13 Connections
cp.curActConns = 66
Returning 3 Connections
cp.curActConns = 69
Returning 3 Connections
Allocating 5 Connections
cp.curActConns = 71
Returning 1 Connections
Allocating 10 Connections
cp.curActConns = 78
Returning 0 Connections
cp.curActConns = 80
Returning 1 Connections
Allocating 16 Connections
cp.curActConns = 90
Returning 3 Connections
cp.curActConns = 92
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 93
Returning 3 Connections
Allocating 21 Connections
cp.curActConns = 101
Returning 3 Connections
cp.curActConns = 108
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 109
Returning 3 Connections
Allocating 22 Connections
cp.curActConns = 112
Returning 4 Connections
cp.curActConns = 124
Returning 2 Connections
Allocating 13 Connections
cp.curActConns = 124
Returning 1 Connections
cp.curActConns = 134
Returning 0 Connections
Allocating 23 Connections
cp.curActConns = 136
Returning 3 Connections
cp.curActConns = 148
Returning 2 Connections
cp.curActConns = 152
Returning 3 Connections
Allocating 16 Connections
cp.curActConns = 160
Returning 4 Connections
cp.curActConns = 161
Allocating 8 Connections
Returning 3 Connections
cp.curActConns = 166
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 167
Returning 3 Connections
Allocating 11 Connections
cp.curActConns = 171
Returning 1 Connections
cp.curActConns = 174
Returning 2 Connections
Allocating 15 Connections
cp.curActConns = 182
Returning 3 Connections
cp.curActConns = 184
Returning 2 Connections
Allocating 18 Connections
cp.curActConns = 193
Returning 0 Connections
cp.curActConns = 200
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 197
Returning 1 Connections
Allocating 15 Connections
cp.curActConns = 205
Returning 2 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 2 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 210
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 208
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 2 Connections
cp.curActConns = 212
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 215
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 214
Returning 3 Connections
Allocating 1 Connections
cp.curActConns = 212
Returning 4 Connections
Allocating 1 Connections
cp.curActConns = 209
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 207
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 209
Returning 4 Connections
Allocating 3 Connections
cp.curActConns = 208
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 206
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 209
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 1 Connections
Allocating 1 Connections
cp.curActConns = 210
Returning 0 Connections
Allocating 1 Connections
cp.curActConns = 211
Returning 4 Connections
Allocating 4 Connections
cp.curActConns = 211
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 213
Returning 3 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 215
Returning 2 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 3 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 215
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 215
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 213
Returning 2 Connections
Allocating 0 Connections
cp.curActConns = 211
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 214
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 216
Returning 0 Connections
cp.curActConns = 217
Allocating 4 Connections
Returning 0 Connections
cp.curActConns = 221
Returning 3 Connections
Allocating 0 Connections
cp.curActConns = 218
Returning 4 Connections
Allocating 2 Connections
cp.curActConns = 216
Returning 0 Connections
Allocating 0 Connections
cp.curActConns = 216
Returning 2 Connections
Allocating 4 Connections
cp.curActConns = 218
Returning 0 Connections
Allocating 3 Connections
cp.curActConns = 221
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 220
Returning 1 Connections
Allocating 0 Connections
cp.curActConns = 219
Returning 1 Connections
Allocating 4 Connections
cp.curActConns = 222
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 221
Returning 2 Connections
Allocating 1 Connections
cp.curActConns = 220
Returning 0 Connections
Allocating 4 Connections
cp.curActConns = 224
Returning 2 Connections
Allocating 2 Connections
cp.curActConns = 224
Retuning from startDeallocatorRoutine
Retuning from startAllocatorRoutine
2022-09-02T03:19:16.745+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:19:17.712+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestSustainedHighConns (59.07s)
=== RUN TestLowWM
2022-09-02T03:19:20.746+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 20 overflow 5 low WM 10 relConn batch size 2 ...
2022-09-02T03:20:20.762+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-02T03:21:20.778+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] active conns 0, free conns 10
2022-09-02T03:21:26.253+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:21:26.780+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestLowWM (129.51s)
=== RUN TestTotalConns
2022-09-02T03:21:30.255+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 120 overflow 5 low WM 10 relConn batch size 10 ...
2022-09-02T03:21:44.430+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:21:45.261+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestTotalConns (18.18s)
=== RUN TestUpdateTickRate
2022-09-02T03:21:48.431+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] started poolsize 40 overflow 5 low WM 2 relConn batch size 2 ...
2022-09-02T03:22:09.279+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] ... stopped
2022-09-02T03:22:09.443+05:30 [Info] [Queryport-connpool:127.0.0.1:15151] Stopping releaseConnsRoutine
--- PASS: TestUpdateTickRate (24.85s)
PASS
ok github.com/couchbase/indexing/secondary/queryport/client 322.836s
Starting server: attempt 1
Functional tests
2022/09/02 03:24:21 In TestMain()
2022/09/02 03:24:21 otp node fetch error: json: cannot unmarshal string into Go value of type couchbase.Pool
2022/09/02 03:24:21 Initialising services with role: kv,n1ql on node: 127.0.0.1:9000
2022/09/02 03:24:22 Initialising web UI on node: 127.0.0.1:9000
2022/09/02 03:24:22 InitWebCreds, response is: {"newBaseUri":"http://127.0.0.1:9000/"}
2022/09/02 03:24:23 Setting data quota of 1500M and Index quota of 1500M
2022/09/02 03:24:24 Adding node: https://127.0.0.1:19001 with role: kv,index to the cluster
2022/09/02 03:24:32 AddNode: Successfully added node: 127.0.0.1:9001 (role kv,index), response: {"otpNode":"n_1@127.0.0.1"}
2022/09/02 03:24:37 Rebalance progress: 0
2022/09/02 03:24:42 Rebalance progress: 0
2022/09/02 03:24:47 Rebalance progress: 0
2022/09/02 03:24:52 Rebalance progress: 0
2022/09/02 03:24:57 Rebalance progress: 0
2022/09/02 03:25:02 Rebalance progress: 0
2022/09/02 03:25:07 Rebalance progress: 0
2022/09/02 03:25:12 Rebalance progress: 0
2022/09/02 03:25:17 Rebalance progress: 0
2022/09/02 03:25:22 Rebalance progress: 0
2022/09/02 03:25:27 Rebalance progress: 0
2022/09/02 03:25:32 Rebalance progress: 0
2022/09/02 03:25:37 Rebalance progress: 0
2022/09/02 03:25:42 Rebalance failed. See logs for detailed reason. You can try again.
2022/09/02 03:25:42 Error while initialising cluster: AddNodeAndRebalance: Error during rebalance, err: Rebalance failed
panic: Error while initialising cluster: AddNodeAndRebalance: Error during rebalance, err: Rebalance failed
goroutine 1 [running]:
panic({0xe8aee0, 0xc000207190})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/panic.go:941 +0x397 fp=0xc000549ca8 sp=0xc000549be8 pc=0x43b757
log.Panicf({0x1025e70?, 0x20?}, {0xc000549df8?, 0xd?, 0xc00003cf50?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/log/log.go:392 +0x67 fp=0xc000549cf0 sp=0xc000549ca8 pc=0x5bdfc7
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/functionaltests.TestMain(0x4461b1?)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:94 +0x456 fp=0xc000549ec8 sp=0xc000549cf0 pc=0xd31a76
main.main()
_testmain.go:483 +0x1d3 fp=0xc000549f80 sp=0xc000549ec8 pc=0xe0cfd3
runtime.main()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:250 +0x212 fp=0xc000549fe0 sp=0xc000549f80 pc=0x43e2d2
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x46ec21
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060fb0 sp=0xc000060f90 pc=0x43e696
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.forcegchelper()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:301 +0xad fp=0xc000060fe0 sp=0xc000060fb0 pc=0x43e52d
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46ec21
created by runtime.init.6
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:289 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c790 sp=0xc00005c770 pc=0x43e696
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005c7c8 sp=0xc00005c790 pc=0x4297f7
runtime.gcenable.func1()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005c7e0 sp=0xc00005c7c8 pc=0x41f3a6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46ec21
created by runtime.gcenable
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0x3a9aa9faa4a5?, 0x10000?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cf20 sp=0xc00005cf00 pc=0x43e696
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005cfc8 sp=0xc00005cf20 pc=0x427605
runtime.gcenable.func2()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005cfe0 sp=0xc00005cfc8 pc=0x41f346
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46ec21
created by runtime.gcenable
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0xaa
goroutine 3 [finalizer wait]:
runtime.gopark(0x0?, 0x109fd68?, 0x60?, 0x0?, 0x2000000020?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060630 sp=0xc000060610 pc=0x43e696
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.runfinq()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000607e0 sp=0xc000060630 pc=0x41e453
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46ec21
created by runtime.createfing
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:157 +0x45
goroutine 34 [select]:
runtime.gopark(0xc000061798?, 0x2?, 0x47?, 0xe7?, 0xc00006178c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061618 sp=0xc0000615f8 pc=0x43e696
runtime.selectgo(0xc000061798, 0xc000061788, 0x0?, 0x0, 0xb?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061758 sp=0xc000061618 pc=0x44e112
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc0002040f0)
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:396 +0x67 fp=0xc0000617c8 sp=0xc000061758 pc=0x785e07
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x26 fp=0xc0000617e0 sp=0xc0000617c8 pc=0x786a86
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46ec21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x37a
goroutine 35 [select]:
runtime.gopark(0xc000244798?, 0x2?, 0x0?, 0x0?, 0xc00024478c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000244608 sp=0xc0002445e8 pc=0x43e696
runtime.selectgo(0xc000244798, 0xc000244788, 0x0?, 0x0, 0x0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000244748 sp=0xc000244608 pc=0x44e112
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc000204108)
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:316 +0x85 fp=0xc0002447c8 sp=0xc000244748 pc=0x785825
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x26 fp=0xc0002447e0 sp=0xc0002447c8 pc=0x786a26
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002447e8 sp=0xc0002447e0 pc=0x46ec21
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x3ca
goroutine 36 [IO wait]:
runtime.gopark(0xc00022c340?, 0xc00004e000?, 0x70?, 0x38?, 0x484542?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000543800 sp=0xc0005437e0 pc=0x43e696
runtime.netpollblock(0xc000145000?, 0x1000?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000543838 sp=0xc000543800 pc=0x437137
internal/poll.runtime_pollWait(0x7fc6c0444fc0, 0x72)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000543858 sp=0xc000543838 pc=0x469209
internal/poll.(*pollDesc).wait(0xc000124100?, 0xc000145000?, 0x0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000543880 sp=0xc000543858 pc=0x4a2132
internal/poll.(*pollDesc).waitRead(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000124100, {0xc000145000, 0x1000, 0x1000})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000543900 sp=0xc000543880 pc=0x4a349a
net.(*netFD).Read(0xc000124100, {0xc000145000?, 0xc00051e990?, 0xc0005439d0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000543948 sp=0xc000543900 pc=0x669209
net.(*conn).Read(0xc00011a010, {0xc000145000?, 0x11?, 0xc000543a68?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000543990 sp=0xc000543948 pc=0x679485
bufio.(*Reader).Read(0xc0001141e0, {0xc000274001, 0x5ff, 0x453934?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc0005439c8 sp=0xc000543990 pc=0x5206f4
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x191?, {0xc000274001?, 0x8?, 0x20?})
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc0005439f8 sp=0xc0005439c8 pc=0x7d7045
encoding/json.(*Decoder).refill(0xc000225400)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000543a48 sp=0xc0005439f8 pc=0x565bbf
encoding/json.(*Decoder).readValue(0xc000225400)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000543a98 sp=0xc000543a48 pc=0x5657bb
encoding/json.(*Decoder).Decode(0xc000225400, {0xebbc00, 0xc000200380})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000543ac8 sp=0xc000543a98 pc=0x565418
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc000200360, 0xc00020e4e0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc000543b08 sp=0xc000543ac8 pc=0x7d62a5
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc000216690?, 0x4cd388?)
:1 +0x2a fp=0xc000543b28 sp=0xc000543b08 pc=0x7d8dca
net/rpc.(*Server).readRequestHeader(0xc000216690, {0x11b5fe8, 0xc000206610})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:587 +0x66 fp=0xc000543bf8 sp=0xc000543b28 pc=0x7d59c6
net/rpc.(*Server).readRequest(0x0?, {0x11b5fe8, 0xc000206610})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:547 +0x3b fp=0xc000543cd0 sp=0xc000543bf8 pc=0x7d551b
net/rpc.(*Server).ServeCodec(0xc000216690, {0x11b5fe8?, 0xc000206610})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:462 +0x87 fp=0xc000543dc8 sp=0xc000543cd0 pc=0x7d4c47
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000244f60?, 0xc00012cfa0)
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc000543f38 sp=0xc000543dc8 pc=0x7d7799
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0x11ac700?, 0xc000122030?})
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc000543f70 sp=0xc000543f38 pc=0x7d7e98
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc00023e000)
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc000543fc0 sp=0xc000543f70 pc=0x7e1c1d
github.com/couchbase/cbauth.startDefault.func1()
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc000543fe0 sp=0xc000543fc0 pc=0x7e1f05
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x46ec21
created by github.com/couchbase/cbauth.startDefault
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9
goroutine 11 [select]:
runtime.gopark(0xc000128f90?, 0x2?, 0xd8?, 0x8d?, 0xc000128f24?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000128d90 sp=0xc000128d70 pc=0x43e696
runtime.selectgo(0xc000128f90, 0xc000128f20, 0xc0004a27c0?, 0x0, 0xc00049fbc0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000128ed0 sp=0xc000128d90 pc=0x44e112
net/http.(*persistConn).writeLoop(0xc00054a000)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000128fc8 sp=0xc000128ed0 pc=0x76d1b5
net/http.(*Transport).dialConn.func6()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000128fe0 sp=0xc000128fc8 pc=0x769c46
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000128fe8 sp=0xc000128fe0 pc=0x46ec21
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791
goroutine 10 [IO wait]:
runtime.gopark(0xc0004a81a0?, 0xc000052a00?, 0x68?, 0x5b?, 0x484542?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000075af8 sp=0xc000075ad8 pc=0x43e696
runtime.netpollblock(0xc0004ac000?, 0x1000?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000075b30 sp=0xc000075af8 pc=0x437137
internal/poll.runtime_pollWait(0x7fc6c0444ed0, 0x72)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000075b50 sp=0xc000075b30 pc=0x469209
internal/poll.(*pollDesc).wait(0xc000540180?, 0xc0004ac000?, 0x0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000075b78 sp=0xc000075b50 pc=0x4a2132
internal/poll.(*pollDesc).waitRead(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000540180, {0xc0004ac000, 0x1000, 0x1000})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000075bf8 sp=0xc000075b78 pc=0x4a349a
net.(*netFD).Read(0xc000540180, {0xc0004ac000?, 0x40bae9?, 0x4?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000075c40 sp=0xc000075bf8 pc=0x669209
net.(*conn).Read(0xc000010888, {0xc0004ac000?, 0x0?, 0x0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000075c88 sp=0xc000075c40 pc=0x679485
net/http.(*persistConn).Read(0xc00054a000, {0xc0004ac000?, 0xc0000466c0?, 0xc000075d30?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000075ce8 sp=0xc000075c88 pc=0x76a6ae
bufio.(*Reader).fill(0xc0001e9440)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000075d20 sp=0xc000075ce8 pc=0x520123
bufio.(*Reader).Peek(0xc0001e9440, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:144 +0x5d fp=0xc000075d40 sp=0xc000075d20 pc=0x52027d
net/http.(*persistConn).readLoop(0xc00054a000)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2093 +0x1ac fp=0xc000075fc8 sp=0xc000075d40 pc=0x76b4cc
net/http.(*Transport).dialConn.func5()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000075fe0 sp=0xc000075fc8 pc=0x769ca6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x46ec21
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e
goroutine 66 [GC worker (idle)]:
runtime.gopark(0xe5da60?, 0xc00003d5e7?, 0x16?, 0x0?, 0x11b5fe8?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000240758 sp=0xc000240738 pc=0x43e696
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0002407e0 sp=0xc000240758 pc=0x421485
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0002407e8 sp=0xc0002407e0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x3a9aa9df1640?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000240f58 sp=0xc000240f38 pc=0x43e696
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000240fe0 sp=0xc000240f58 pc=0x421485
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000240fe8 sp=0xc000240fe0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0xe5da60?, 0xc00050c317?, 0x16?, 0x0?, 0x11b5fe8?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062758 sp=0xc000062738 pc=0x43e696
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0000627e0 sp=0xc000062758 pc=0x421485
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0xc000244fa0?, 0xc000244fb0?, 0x1d?, 0x1c?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000244f58 sp=0xc000244f38 pc=0x43e696
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc000244fe0 sp=0xc000244f58 pc=0x421485
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000244fe8 sp=0xc000244fe0 pc=0x46ec21
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
signal: aborted (core dumped)
FAIL github.com/couchbase/indexing/secondary/tests/functionaltests 81.003s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_functests_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
2022/09/02 03:25:44 In TestMain()
2022/09/02 03:25:44 ChangeIndexerSettings: Host Port 0 Nodes []
2022/09/02 03:25:44 Changing config key indexer.api.enableTestServer to value true
2022/09/02 03:25:44 Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
panic: Error in ChangeIndexerSettings: Post "http://:2/internal/settings": dial tcp :2: connect: connection refused
goroutine 1 [running]:
panic({0xcd25e0, 0xc00050c9a0})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/panic.go:941 +0x397 fp=0xc0001d3d40 sp=0xc0001d3c80 pc=0x43a6d7
log.Panicf({0xe3f566?, 0x1e?}, {0xc0001d3e38?, 0x1c?, 0xcc7ce0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/log/log.go:392 +0x67 fp=0xc0001d3d88 sp=0xc0001d3d40 pc=0x5bb407
github.com/couchbase/indexing/secondary/tests/framework/common.HandleError(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/framework/common/util.go:76
github.com/couchbase/indexing/secondary/tests/largedatatests.TestMain(0x445131?)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/tests/largedatatests/common_test.go:52 +0x468 fp=0xc0001d3ec8 sp=0xc0001d3d88 pc=0xc5e768
main.main()
_testmain.go:59 +0x1d3 fp=0xc0001d3f80 sp=0xc0001d3ec8 pc=0xc64833
runtime.main()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:250 +0x212 fp=0xc0001d3fe0 sp=0xc0001d3f80 pc=0x43d252
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001d3fe8 sp=0xc0001d3fe0 pc=0x46dba1
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060fb0 sp=0xc000060f90 pc=0x43d616
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.forcegchelper()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:301 +0xad fp=0xc000060fe0 sp=0xc000060fb0 pc=0x43d4ad
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x46dba1
created by runtime.init.6
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:289 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005c790 sp=0xc00005c770 pc=0x43d616
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgsweep(0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005c7c8 sp=0xc00005c790 pc=0x428777
runtime.gcenable.func1()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x26 fp=0xc00005c7e0 sp=0xc00005c7c8 pc=0x41e326
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005c7e8 sp=0xc00005c7e0 pc=0x46dba1
created by runtime.gcenable
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:177 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0x3aab86428e9e?, 0x10000?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005cf20 sp=0xc00005cf00 pc=0x43d616
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.bgscavenge(0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgcscavenge.go:364 +0x2a5 fp=0xc00005cfc8 sp=0xc00005cf20 pc=0x426585
runtime.gcenable.func2()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0x26 fp=0xc00005cfe0 sp=0xc00005cfc8 pc=0x41e2c6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005cfe8 sp=0xc00005cfe0 pc=0x46dba1
created by runtime.gcenable
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:178 +0xaa
goroutine 20 [finalizer wait]:
runtime.gopark(0x0?, 0xea3598?, 0xc0?, 0x40?, 0x2000000020?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000060630 sp=0xc000060610 pc=0x43d616
runtime.goparkunlock(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:367
runtime.runfinq()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:177 +0xb3 fp=0xc0000607e0 sp=0xc000060630 pc=0x41d3d3
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x46dba1
created by runtime.createfing
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mfinal.go:157 +0x45
goroutine 3 [select]:
runtime.gopark(0xc000061798?, 0x2?, 0x0?, 0x0?, 0xc00006178c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061618 sp=0xc0000615f8 pc=0x43d616
runtime.selectgo(0xc000061798, 0xc000061788, 0x0?, 0x0, 0x0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061758 sp=0xc000061618 pc=0x44d092
github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0xc00000e0f0)
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:396 +0x67 fp=0xc0000617c8 sp=0xc000061758 pc=0x779647
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func2()
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x26 fp=0xc0000617e0 sp=0xc0000617c8 pc=0x77a2c6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x46dba1
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:561 +0x37a
goroutine 4 [select]:
runtime.gopark(0xc000061f98?, 0x2?, 0x0?, 0x0?, 0xc000061f8c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000061e08 sp=0xc000061de8 pc=0x43d616
runtime.selectgo(0xc000061f98, 0xc000061f88, 0x0?, 0x0, 0x0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000061f48 sp=0xc000061e08 pc=0x44d092
github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0xc00000e108)
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:316 +0x85 fp=0xc000061fc8 sp=0xc000061f48 pc=0x779065
github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest.func3()
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x26 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x77a266
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x46dba1
created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest
/opt/build/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:562 +0x3ca
goroutine 5 [IO wait]:
runtime.gopark(0xc0000036c0?, 0xc00004e000?, 0x70?, 0x98?, 0x483482?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00015d800 sp=0xc00015d7e0 pc=0x43d616
runtime.netpollblock(0xc0001a9000?, 0x1000?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc00015d838 sp=0xc00015d800 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6b0d8, 0x72)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc00015d858 sp=0xc00015d838 pc=0x468189
internal/poll.(*pollDesc).wait(0xc00012c180?, 0xc0001a9000?, 0x0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc00015d880 sp=0xc00015d858 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc00012c180, {0xc0001a9000, 0x1000, 0x1000})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00015d900 sp=0xc00015d880 pc=0x4a211a
net.(*netFD).Read(0xc00012c180, {0xc0001a9000?, 0xc0004743f0?, 0xc?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc00015d948 sp=0xc00015d900 pc=0x665589
net.(*conn).Read(0xc00012a4f8, {0xc0001a9000?, 0x11?, 0x0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc00015d990 sp=0xc00015d948 pc=0x674aa5
bufio.(*Reader).Read(0xc0001144e0, {0xc000038601, 0x5ff, 0x4528b4?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:236 +0x1b4 fp=0xc00015d9c8 sp=0xc00015d990 pc=0x51dd14
github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x203000?, {0xc000038601?, 0x203000?, 0xc0004327e0?})
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:102 +0x25 fp=0xc00015d9f8 sp=0xc00015d9c8 pc=0x7b9da5
encoding/json.(*Decoder).refill(0xc000480280)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc00015da48 sp=0xc00015d9f8 pc=0x562fff
encoding/json.(*Decoder).readValue(0xc000480280)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc00015da98 sp=0xc00015da48 pc=0x562bfb
encoding/json.(*Decoder).Decode(0xc000480280, {0xcfd5c0, 0xc0003bef80})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc00015dac8 sp=0xc00015da98 pc=0x562858
net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0xc0003bef60, 0xc0004327e0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/jsonrpc/server.go:66 +0x85 fp=0xc00015db08 sp=0xc00015dac8 pc=0x7b9005
github.com/couchbase/cbauth/revrpc.(*jsonServerCodec).ReadRequestHeader(0xc000099900?, 0x4cc008?)
:1 +0x2a fp=0xc00015db28 sp=0xc00015db08 pc=0x7bbb2a
net/rpc.(*Server).readRequestHeader(0xc000099900, {0xf8d6e8, 0xc000474370})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:587 +0x66 fp=0xc00015dbf8 sp=0xc00015db28 pc=0x7b8726
net/rpc.(*Server).readRequest(0x0?, {0xf8d6e8, 0xc000474370})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:547 +0x3b fp=0xc00015dcd0 sp=0xc00015dbf8 pc=0x7b827b
net/rpc.(*Server).ServeCodec(0xc000099900, {0xf8d6e8?, 0xc000474370})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/rpc/server.go:462 +0x87 fp=0xc00015ddc8 sp=0xc00015dcd0 pc=0x7b79a7
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc000062760?, 0xc00006efa0)
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:192 +0x5d9 fp=0xc00015df38 sp=0xc00015ddc8 pc=0x7ba4f9
github.com/couchbase/cbauth/revrpc.BabysitService(0x0?, 0x0?, {0xf84480?, 0xc000136618?})
/opt/build/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:288 +0x58 fp=0xc00015df70 sp=0xc00015df38 pc=0x7babf8
github.com/couchbase/cbauth.runRPCForSvc(0x0?, 0xc0000b2000)
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xbd fp=0xc00015dfc0 sp=0xc00015df70 pc=0x7c44fd
github.com/couchbase/cbauth.startDefault.func1()
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x25 fp=0xc00015dfe0 sp=0xc00015dfc0 pc=0x7c47e5
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00015dfe8 sp=0xc00015dfe0 pc=0x46dba1
created by github.com/couchbase/cbauth.startDefault
/opt/build/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xf9
goroutine 6 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062e78 sp=0xc000062e58 pc=0x43d616
runtime.chanrecv(0xc0003beea0, 0xc000062f90, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc000062f08 sp=0xc000062e78 pc=0x40b5cc
runtime.chanrecv2(0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:445 +0x18 fp=0xc000062f30 sp=0xc000062f08 pc=0x40b038
github.com/couchbase/goutils/systemeventlog.(*SystemEventLoggerImpl).logEvents(0xc0000d9880)
/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:186 +0xb7 fp=0xc000062fc8 sp=0xc000062f30 pc=0xb001b7
github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger.func1()
/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x26 fp=0xc000062fe0 sp=0xc000062fc8 pc=0xaffac6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x46dba1
created by github.com/couchbase/goutils/systemeventlog.NewSystemEventLogger
/opt/build/goproj/src/github.com/couchbase/goutils/systemeventlog/system_event_logger.go:125 +0x1d6
goroutine 8 [select]:
runtime.gopark(0xc0001d9ba0?, 0x3?, 0x0?, 0x30?, 0xc0001d9b1a?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001d9978 sp=0xc0001d9958 pc=0x43d616
runtime.selectgo(0xc0001d9ba0, 0xc0001d9b14, 0x3?, 0x0, 0x0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001d9ab8 sp=0xc0001d9978 pc=0x44d092
github.com/couchbase/cbauth/metakv.doRunObserveChildren(0xc000076840?, {0xe504e2, 0x1b}, 0xc0001d9e68, 0xc000042600)
/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:301 +0x429 fp=0xc0001d9e40 sp=0xc0001d9ab8 pc=0x9b8289
github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...)
/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:259
github.com/couchbase/cbauth/metakv.RunObserveChildren({0xe504e2?, 0x0?}, 0x0?, 0x0?)
/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:389 +0x58 fp=0xc0001d9e88 sp=0xc0001d9e40 pc=0x9b8838
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0?, {0x0?, 0x0?})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1579 +0xc7 fp=0xc0001d9f00 sp=0xc0001d9e88 pc=0xb05147
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc0001d9fa0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc0001d9f38 sp=0xc0001d9f00 pc=0xace643
github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1584 +0xdf fp=0xc0001d9fe0 sp=0xc0001d9f38 pc=0xb04fff
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001d9fe8 sp=0xc0001d9fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1572 +0xaf
goroutine 15 [IO wait]:
runtime.gopark(0xc0002381a0?, 0xc00004e000?, 0x68?, 0x2b?, 0x483482?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000202af8 sp=0xc000202ad8 pc=0x43d616
runtime.netpollblock(0xc0004f6000?, 0x1000?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc000202b30 sp=0xc000202af8 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6afe8, 0x72)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc000202b50 sp=0xc000202b30 pc=0x468189
internal/poll.(*pollDesc).wait(0xc0004da280?, 0xc0004f6000?, 0x0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000202b78 sp=0xc000202b50 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004da280, {0xc0004f6000, 0x1000, 0x1000})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000202bf8 sp=0xc000202b78 pc=0x4a211a
net.(*netFD).Read(0xc0004da280, {0xc0004f6000?, 0x8?, 0xc000560c88?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000202c40 sp=0xc000202bf8 pc=0x665589
net.(*conn).Read(0xc00020e098, {0xc0004f6000?, 0xc000010038?, 0xc00003d2c0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000202c88 sp=0xc000202c40 pc=0x674aa5
net/http.(*persistConn).Read(0xc00022e7e0, {0xc0004f6000?, 0x40a1fd?, 0x60?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000202ce8 sp=0xc000202c88 pc=0x76588e
bufio.(*Reader).fill(0xc000201080)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000202d20 sp=0xc000202ce8 pc=0x51d743
bufio.(*Reader).Peek(0xc000201080, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:144 +0x5d fp=0xc000202d40 sp=0xc000202d20 pc=0x51d89d
net/http.(*persistConn).readLoop(0xc00022e7e0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2093 +0x1ac fp=0xc000202fc8 sp=0xc000202d40 pc=0x7666ac
net/http.(*Transport).dialConn.func5()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000202fe0 sp=0xc000202fc8 pc=0x764e86
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000202fe8 sp=0xc000202fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e
goroutine 22 [GC worker (idle)]:
runtime.gopark(0xc0000627a0?, 0xc0000627b0?, 0xfd?, 0x44?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000062758 sp=0xc000062738 pc=0x43d616
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc0000627e0 sp=0xc000062758 pc=0x420405
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x3aab863b9083?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005df58 sp=0xc00005df38 pc=0x43d616
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005dfe0 sp=0xc00005df58 pc=0x420405
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x3aab863b5e3d?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005e758 sp=0xc00005e738 pc=0x43d616
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005e7e0 sp=0xc00005e758 pc=0x420405
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x3aab863d292a?, 0x0?, 0x0?, 0x0?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005ef58 sp=0xc00005ef38 pc=0x43d616
runtime.gcBgMarkWorker()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1207 +0xe5 fp=0xc00005efe0 sp=0xc00005ef58 pc=0x420405
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x46dba1
created by runtime.gcBgMarkStartWorkers
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/mgc.go:1131 +0x25
goroutine 53 [runnable]:
runtime.gopark(0xc0001dc9c0?, 0x6?, 0x80?, 0xc6?, 0xc0001dc7d4?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001dc638 sp=0xc0001dc618 pc=0x43d616
runtime.selectgo(0xc0001dc9c0, 0xc0001dc7c8, 0xe45617?, 0x0, 0xc0001dc7b0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001dc778 sp=0xc0001dc638 pc=0x44d092
net/http.(*persistConn).roundTrip(0xc00043aea0, 0xc000214300)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2620 +0x974 fp=0xc0001dca30 sp=0xc0001dc778 pc=0x769254
net/http.(*Transport).roundTrip(0x170e3e0, 0xc000226800)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:594 +0x7c9 fp=0xc0001dcc68 sp=0xc0001dca30 pc=0x75cce9
net/http.(*Transport).RoundTrip(0xc000226800?, 0xf83c00?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc0001dcc88 sp=0xc0001dcc68 pc=0x744f19
net/http.send(0xc000226700, {0xf83c00, 0x170e3e0}, {0xdceb60?, 0x48e901?, 0x1787d40?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:252 +0x5d8 fp=0xc0001dce68 sp=0xc0001dcc88 pc=0x706818
net/http.(*Client).send(0xc00020a9c0, 0xc000226700, {0xc0002284c0?, 0xc0001dcf70?, 0x1787d40?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:176 +0x9b fp=0xc0001dcee0 sp=0xc0001dce68 pc=0x7060bb
net/http.(*Client).do(0xc00020a9c0, 0xc000226700)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:725 +0x8f5 fp=0xc0001dd0e0 sp=0xc0001dcee0 pc=0x7084f5
net/http.(*Client).Do(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc00021e9c0?, 0x3b?}, 0xc0001dd470, {0x0, 0x0}, 0x0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0001dd1e0 sp=0xc0001dd0e0 pc=0x8626c9
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc0001de000, {0xc000248510, 0x26}, {0xc0002101c0?, 0xc0001dd690?}, {0xcb0740, 0xc000218378}, 0xcb6bc0?)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc0001dd428 sp=0xc0001dd1e0 pc=0x884093
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.(*Pool).GetServerGroups(0xc000144878)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:907 +0xbd fp=0xc0001dd498 sp=0xc0001dd428 pc=0x88837d
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchServerGroups(0xc000144780)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:694 +0x4e fp=0xc0001dd778 sp=0xc0001dd498 pc=0xab06ce
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x7f7a7bb1c5b8?, {0x0?, 0x0?})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:381 +0x62f fp=0xc0001dde78 sp=0xc0001dd778 pc=0xaad12f
github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0xc0001ddec8)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x83 fp=0xc0001ddeb0 sp=0xc0001dde78 pc=0xace643
github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0xc00003d2c0?)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:402 +0x6b fp=0xc0001ddf00 sp=0xc0001ddeb0 pc=0xaacacb
github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache({0xc00003d2c0?, 0x0?}, {0xe3fad7, 0x7}, {0xe4b8b8, 0x16})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:138 +0x15b fp=0xc0001ddf40 sp=0xc0001ddf00 pc=0xaab9db
github.com/couchbase/indexing/secondary/common.MonitorInternalVersion(0x0?, {0xe3e4ae, 0x5}, {0xc00003d2c0, 0xe})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/internal_version.go:791 +0x5b fp=0xc0001ddfa8 sp=0xc0001ddf40 pc=0xacc1fb
github.com/couchbase/indexing/secondary/queryport/client.watchClusterVer.func2()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:2046 +0x3b fp=0xc0001ddfe0 sp=0xc0001ddfa8 pc=0xc257fb
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001ddfe8 sp=0xc0001ddfe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.watchClusterVer
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:2046 +0xd5
goroutine 54 [select]:
runtime.gopark(0xc000565f98?, 0x2?, 0x0?, 0x30?, 0xc000565f8c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000565e18 sp=0xc000565df8 pc=0x43d616
runtime.selectgo(0xc000565f98, 0xc000565f88, 0x0?, 0x0, 0x0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000565f58 sp=0xc000565e18 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*schedTokenMonitor).updater(0xc0001161e0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2389 +0x92 fp=0xc000565fc8 sp=0xc000565f58 pc=0xc37f92
github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor.func1()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x26 fp=0xc000565fe0 sp=0xc000565fc8 pc=0xc368c6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newSchedTokenMonitor
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2186 +0x2e5
goroutine 57 [select]:
runtime.gopark(0xc0001daea8?, 0x6?, 0x0?, 0x30?, 0xc0001dacbc?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0001dab20 sp=0xc0001dab00 pc=0x43d616
runtime.selectgo(0xc0001daea8, 0xc0001dacb0, 0xe45617?, 0x0, 0xc0001daef0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc0001dac60 sp=0xc0001dab20 pc=0x44d092
net/http.(*persistConn).roundTrip(0xc00022e7e0, 0xc0002143c0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2620 +0x974 fp=0xc0001daf18 sp=0xc0001dac60 pc=0x769254
net/http.(*Transport).roundTrip(0x170e3e0, 0xc000226a00)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:594 +0x7c9 fp=0xc0001db150 sp=0xc0001daf18 pc=0x75cce9
net/http.(*Transport).RoundTrip(0xc000226a00?, 0xf83c00?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/roundtrip.go:17 +0x19 fp=0xc0001db170 sp=0xc0001db150 pc=0x744f19
net/http.send(0xc000226900, {0xf83c00, 0x170e3e0}, {0xdceb60?, 0x48e901?, 0x1787d40?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:252 +0x5d8 fp=0xc0001db350 sp=0xc0001db170 pc=0x706818
net/http.(*Client).send(0xc00020ab70, 0xc000226900, {0x7f7a7ba5e868?, 0x150?, 0x1787d40?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:176 +0x9b fp=0xc0001db3c8 sp=0xc0001db350 pc=0x7060bb
net/http.(*Client).do(0xc00020ab70, 0xc000226900)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:725 +0x8f5 fp=0xc0001db5c8 sp=0xc0001db3c8 pc=0x7084f5
net/http.(*Client).Do(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/client.go:593
github.com/couchbase/indexing/secondary/security.getWithAuthInternal({0xc000216540?, 0x1b?}, 0xc0001db960, {0x0, 0x0}, 0x0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:669 +0x549 fp=0xc0001db6c8 sp=0xc0001db5c8 pc=0x8626c9
github.com/couchbase/indexing/secondary/security.GetWithAuthNonTLS(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:604
github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0xc000222a20, {0xe3ebec, 0x6}, {0xd43bc0?, 0xc000534001?}, {0xcb0680, 0xc0004da198}, 0xc00021ec00?)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:359 +0x1b3 fp=0xc0001db910 sp=0xc0001db6c8 pc=0x884093
github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:530
github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth({0xc00021ec00, 0x3f}, {0xf83820?, 0xc000210260})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:594 +0x125 fp=0xc0001db988 sp=0xc0001db910 pc=0x886045
github.com/couchbase/indexing/secondary/dcp.Connect({0xc00021ec00, 0x3f})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:600 +0xce fp=0xc0001dbb20 sp=0xc0001db988 pc=0x8861ee
github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier({0xc00021ec00, 0x3f}, {0xe3fad7, 0x7}, {0xe42d8c, 0xb})
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:239 +0x1a8 fp=0xc0001dbda0 sp=0xc0001dbb20 pc=0xad0f48
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).watchClusterChanges(0xc0000e02c0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:2071 +0xff fp=0xc0001dbfc8 sp=0xc0001dbda0 pc=0xc35c9f
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func1()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x26 fp=0xc0001dbfe0 sp=0xc0001dbfc8 pc=0xc29426
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001dbfe8 sp=0xc0001dbfe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:149 +0x5c5
goroutine 30 [IO wait]:
runtime.gopark(0xc000436000?, 0xc000050500?, 0x10?, 0xa?, 0x483482?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc0000709a0 sp=0xc000070980 pc=0x43d616
runtime.netpollblock(0xc0004dc000?, 0x1000?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:522 +0xf7 fp=0xc0000709d8 sp=0xc0000709a0 pc=0x4360b7
internal/poll.runtime_pollWait(0x7f7a7ba6ad18, 0x72)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/netpoll.go:302 +0x89 fp=0xc0000709f8 sp=0xc0000709d8 pc=0x468189
internal/poll.(*pollDesc).wait(0xc0004da080?, 0xc0004dc000?, 0x0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 fp=0xc000070a20 sp=0xc0000709f8 pc=0x4a0db2
internal/poll.(*pollDesc).waitRead(...)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc0004da080, {0xc0004dc000, 0x1000, 0x1000})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000070aa0 sp=0xc000070a20 pc=0x4a211a
net.(*netFD).Read(0xc0004da080, {0xc0004dc000?, 0x0?, 0x0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/fd_posix.go:55 +0x29 fp=0xc000070ae8 sp=0xc000070aa0 pc=0x665589
net.(*conn).Read(0xc00020e048, {0xc0004dc000?, 0x0?, 0x0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/net.go:183 +0x45 fp=0xc000070b30 sp=0xc000070ae8 pc=0x674aa5
net/http.(*persistConn).Read(0xc00022e360, {0xc0004dc000?, 0x1000?, 0x1000?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1929 +0x4e fp=0xc000070b90 sp=0xc000070b30 pc=0x76588e
bufio.(*Reader).fill(0xc000200ae0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:106 +0x103 fp=0xc000070bc8 sp=0xc000070b90 pc=0x51d743
bufio.(*Reader).ReadSlice(0xc000200ae0, 0x38?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/bufio/bufio.go:371 +0x2f fp=0xc000070c18 sp=0xc000070bc8 pc=0x51e32f
net/http/internal.readChunkLine(0xc0001e8ce8?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:129 +0x25 fp=0xc000070c68 sp=0xc000070c18 pc=0x7036c5
net/http/internal.(*chunkedReader).beginChunk(0xc000124990)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:48 +0x28 fp=0xc000070c98 sp=0xc000070c68 pc=0x703148
net/http/internal.(*chunkedReader).Read(0xc000124990, {0xc000538a00?, 0x0?, 0x0?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/internal/chunked.go:98 +0x14e fp=0xc000070d18 sp=0xc000070c98 pc=0x70340e
net/http.(*body).readLocked(0xc000094640, {0xc000538a00?, 0x73fb7d?, 0xc0001e8e80?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:844 +0x3c fp=0xc000070d68 sp=0xc000070d18 pc=0x75a3fc
net/http.(*body).Read(0x1000000000000?, {0xc000538a00?, 0x0?, 0x7f7a7bb1c5b8?})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transfer.go:836 +0x125 fp=0xc000070de0 sp=0xc000070d68 pc=0x75a2c5
net/http.(*bodyEOFSignal).Read(0xc0000949c0, {0xc000538a00, 0x200, 0x200})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2774 +0x142 fp=0xc000070e60 sp=0xc000070de0 pc=0x769fc2
encoding/json.(*Decoder).refill(0xc0001423c0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:165 +0x17f fp=0xc000070eb0 sp=0xc000070e60 pc=0x562fff
encoding/json.(*Decoder).readValue(0xc0001423c0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:140 +0xbb fp=0xc000070f00 sp=0xc000070eb0 pc=0x562bfb
encoding/json.(*Decoder).Decode(0xc0001423c0, {0xcaee80, 0xc000098320})
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/encoding/json/stream.go:63 +0x78 fp=0xc000070f30 sp=0xc000070f00 pc=0x562858
github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1()
/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x10b fp=0xc000070fe0 sp=0xc000070f30 pc=0x9b872b
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x46dba1
created by github.com/couchbase/cbauth/metakv.doRunObserveChildren
/opt/build/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:280 +0x2eb
goroutine 16 [select]:
runtime.gopark(0xc000207f90?, 0x2?, 0xd8?, 0x7d?, 0xc000207f24?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000207d90 sp=0xc000207d70 pc=0x43d616
runtime.selectgo(0xc000207f90, 0xc000207f20, 0xc000214440?, 0x0, 0xc00020ad20?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000207ed0 sp=0xc000207d90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00022e7e0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000207fc8 sp=0xc000207ed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000207fe0 sp=0xc000207fc8 pc=0x764e26
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000207fe8 sp=0xc000207fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791
goroutine 28 [select]:
runtime.gopark(0xc000206f68?, 0x4?, 0x3?, 0x0?, 0xc000206db0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000206c00 sp=0xc000206be0 pc=0x43d616
runtime.selectgo(0xc000206f68, 0xc000206da8, 0xc000094a80?, 0x0, 0xc000565501?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000206d40 sp=0xc000206c00 pc=0x44d092
net/http.(*persistConn).readLoop(0xc00043aea0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc000206fc8 sp=0xc000206d40 pc=0x7672a5
net/http.(*Transport).dialConn.func5()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000206fe0 sp=0xc000206fc8 pc=0x764e86
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000206fe8 sp=0xc000206fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e
goroutine 29 [select]:
runtime.gopark(0xc00006ff90?, 0x2?, 0xd8?, 0xfd?, 0xc00006ff24?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00006fd90 sp=0xc00006fd70 pc=0x43d616
runtime.selectgo(0xc00006ff90, 0xc00006ff20, 0xc000094480?, 0x0, 0xc00020aa50?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00006fed0 sp=0xc00006fd90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00043aea0)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00006ffc8 sp=0xc00006fed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x764e26
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791
goroutine 11 [select]:
runtime.gopark(0xc000571f68?, 0x4?, 0x3?, 0x0?, 0xc000571db0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000571c00 sp=0xc000571be0 pc=0x43d616
runtime.selectgo(0xc000571f68, 0xc000571da8, 0xc000094640?, 0x0, 0xc000560d01?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000571d40 sp=0xc000571c00 pc=0x44d092
net/http.(*persistConn).readLoop(0xc00022e360)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2213 +0xda5 fp=0xc000571fc8 sp=0xc000571d40 pc=0x7672a5
net/http.(*Transport).dialConn.func5()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x764e86
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1750 +0x173e
goroutine 12 [select]:
runtime.gopark(0xc000209f90?, 0x2?, 0xd8?, 0x9d?, 0xc000209f24?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000209d90 sp=0xc000209d70 pc=0x43d616
runtime.selectgo(0xc000209f90, 0xc000209f20, 0xc000214180?, 0x0, 0xc00020a660?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000209ed0 sp=0xc000209d90 pc=0x44d092
net/http.(*persistConn).writeLoop(0xc00022e360)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:2392 +0xf5 fp=0xc000209fc8 sp=0xc000209ed0 pc=0x768395
net/http.(*Transport).dialConn.func6()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x26 fp=0xc000209fe0 sp=0xc000209fc8 pc=0x764e26
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000209fe8 sp=0xc000209fe0 pc=0x46dba1
created by net/http.(*Transport).dialConn
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/net/http/transport.go:1751 +0x1791
goroutine 58 [chan receive]:
runtime.gopark(0xc00005d6d8?, 0x4431bb?, 0x20?, 0xd7?, 0x459dc5?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005d6c8 sp=0xc00005d6a8 pc=0x43d616
runtime.chanrecv(0xc0003bf560, 0x0, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:577 +0x56c fp=0xc00005d758 sp=0xc00005d6c8 pc=0x40b5cc
runtime.chanrecv1(0xdf8475800?, 0x0?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/chan.go:440 +0x18 fp=0xc00005d780 sp=0xc00005d758 pc=0x40aff8
github.com/couchbase/indexing/secondary/queryport/client.(*metadataClient).logstats(0xc0000e02c0)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:1410 +0x79 fp=0xc00005d7c8 sp=0xc00005d780 pc=0xc30a39
github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient.func2()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x26 fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0xc293c6
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.newMetaBridgeClient
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/meta_client.go:150 +0x605
goroutine 59 [select]:
runtime.gopark(0xc000564f90?, 0x2?, 0x40?, 0xc?, 0xc000564f8c?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc000564e18 sp=0xc000564df8 pc=0x43d616
runtime.selectgo(0xc000564f90, 0xc000564f88, 0x0?, 0x0, 0xf8d598?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc000564f58 sp=0xc000564e18 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).listenMetaChange(0xc000032c00, 0xc0000b0380)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1830 +0x70 fp=0xc000564fc0 sp=0xc000564f58 pc=0xc24930
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func1()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x2a fp=0xc000564fe0 sp=0xc000564fc0 pc=0xc2488a
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000564fe8 sp=0xc000564fe0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1823 +0x28a
goroutine 60 [select]:
runtime.gopark(0xc00005f750?, 0x2?, 0x0?, 0x30?, 0xc00005f714?)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/proc.go:361 +0xd6 fp=0xc00005f590 sp=0xc00005f570 pc=0x43d616
runtime.selectgo(0xc00005f750, 0xc00005f710, 0xe6f1ce?, 0x0, 0xcd1be0?, 0x1)
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/select.go:328 +0x772 fp=0xc00005f6d0 sp=0xc00005f590 pc=0x44d092
github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).logstats(0xc000032c00, 0xc0000b0380)
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1849 +0x238 fp=0xc00005f7c0 sp=0xc00005f6d0 pc=0xc24bb8
github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider.func2()
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2a fp=0xc00005f7e0 sp=0xc00005f7c0 pc=0xc2482a
runtime.goexit()
/home/buildbot/.cbdepscache/exploded/x86_64/go-1.18.4/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x46dba1
created by github.com/couchbase/indexing/secondary/queryport/client.makeWithMetaProvider
/opt/build/goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1824 +0x2ed
signal: aborted (core dumped)
FAIL github.com/couchbase/indexing/secondary/tests/largedatatests 0.141s
Indexer Go routine dump logged in /opt/build/ns_server/logs/n_1/indexer_largedata_pprof.log
curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 127.0.0.1 port 9108 after 1 ms: Connection refused
Integration tests
echo "Running gsi integration tests with 4 node cluster"
Running gsi integration tests with 4 node cluster
scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini conf/simple_gsi_n1ql.conf 1 1 gsi_type=memory_optimized
Printing gsi_type=memory_optimized
gsi_type=memory_optimized
In here
-p makefile=True,gsi_type=memory_optimized
/opt/build/testrunner /opt/build/testrunner
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory ns_dataclean
Built target ns_dataclean
make[1]: Leaving directory '/opt/build/ns_server'
make[1]: Entering directory '/opt/build/ns_server'
cd build && make --no-print-directory all
[ 0%] Built target event_ui_build_prepare
[100%] Built target ns_ui_build_prepare
[100%] Building Go Modules target ns_minify_js using Go 1.18.5
[100%] Built target ns_minify_js
[100%] Building Go Modules target ns_minify_css using Go 1.18.5
[100%] Built target ns_minify_css
[100%] Built target query_ui_build_prepare
[100%] Built target fts_ui_build_prepare
[100%] Built target cbas_ui_build_prepare
[100%] Built target backup_ui_build_prepare
[100%] Built target ui_build
==> enacl (compile)
[100%] Built target enacl
[100%] Built target kv_mappings
[100%] Built target ns_cfg
==> ale (compile)
[100%] Built target ale
==> chronicle (compile)
[100%] Built target chronicle
==> ns_server (compile)
[100%] Built target ns_server
==> gen_smtp (compile)
[100%] Built target gen_smtp
==> ns_babysitter (compile)
[100%] Built target ns_babysitter
==> ns_couchdb (compile)
[100%] Built target ns_couchdb
[100%] Building Go target ns_goport using Go 1.18.5
[100%] Built target ns_goport
[100%] Building Go target ns_generate_cert using Go 1.18.5
[100%] Built target ns_generate_cert
[100%] Building Go target ns_godu using Go 1.18.5
[100%] Built target ns_godu
[100%] Building Go target ns_gosecrets using Go 1.18.5
[100%] Built target ns_gosecrets
[100%] Building Go target ns_generate_hash using Go 1.18.5
[100%] Built target ns_generate_hash
==> chronicle (escriptize)
[100%] Built target chronicle_dump
make[1]: Leaving directory '/opt/build/ns_server'
/opt/build/testrunner
INFO:__main__:Checking arguments...
INFO:__main__:Conf filename: conf/simple_gsi_n1ql.conf
INFO:__main__:Test prefix: gsi.indexscans_gsi.SecondaryIndexingScanTests
INFO:__main__:Test prefix: gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests
INFO:__main__:Test prefix: gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests
INFO:__main__:TestRunner: start...
INFO:__main__:Global Test input params:
INFO:__main__:
Number of tests initially selected before GROUP filters: 11
INFO:__main__:--> Running test: gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
INFO:__main__:Logs folder: /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1
*** TestRunner ***
{'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi',
'conf_file': 'conf/simple_gsi_n1ql.conf',
'gsi_type': 'memory_optimized',
'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini',
'makefile': 'True',
'num_nodes': 4,
'spec': 'simple_gsi_n1ql'}
Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 1, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'False', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1'}
Run before suite setup for gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index
suite_setUp (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... -->before_suite_name:gsi.indexscans_gsi.SecondaryIndexingScanTests.suite_setUp,suite: ]>
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:27 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:27 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused
2022-09-02 03:26:30 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused
2022-09-02 03:26:36 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/pools/default error [Errno 111] Connection refused
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1948-rel-EE-enterprise
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 suite_setUp==============
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] cannot find service node index in cluster
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:48 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:48 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-02 03:26:54 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:54 | INFO | MainProcess | MainThread | [basetestcase.setUp] initializing cluster
2022-09-02 03:26:55 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9000/pools/default with status False: unknown pool
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '22', 'memoryTotal': 15466930176, 'memoryFree': 5484658688, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:55 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:55 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:56 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9001/pools/default with status False: unknown pool
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '23', 'memoryTotal': 15466930176, 'memoryFree': 5313880064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:56 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9001/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:56 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9002/pools/default with status False: unknown pool
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '24', 'memoryTotal': 15466930176, 'memoryFree': 5189267456, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9002/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
http://127.0.0.1:9003/pools/default with status False: unknown pool
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '18', 'memoryTotal': 15466930176, 'memoryFree': 5093285888, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 8184, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:26:57 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] GET http://127.0.0.1:9003/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:asdasd
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-02 03:26:57 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:26:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-02 03:26:58 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [internal_user.delete_user] Exception while deleting user. Exception is -b'"User was not found."'
2022-09-02 03:26:58 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ...
2022-09-02 03:27:03 | INFO | MainProcess | MainThread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [basetestcase.setUp] done initializing cluster
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:27:04 | INFO | MainProcess | MainThread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-02 03:27:04 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-02 03:28:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:05 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 suite_setUp ==============
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:05 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:06 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.277509736964467, 'mem_free': 13542510592, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.setUp] Time to execute basesetup : 102.04294872283936
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-02 03:28:09 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 5 secs. ...
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:14 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:15 | INFO | MainProcess | MainThread | [basetestcase.load] create 2016.0 to default documents...
2022-09-02 03:28:15 | INFO | MainProcess | MainThread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.load] LOAD IS FINISHED
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-02 03:28:17 | INFO | MainProcess | MainThread | [basetestcase.sleep] sleep for 30 secs. ...
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 145.374686ms
2022-09-02 03:28:47 | ERROR | MainProcess | MainThread | [tuq_helper._is_index_in_list] Fail to get index list. List output: {'requestID': 'faed4653-404b-4a47-8aa9-b8380b9a0aca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '145.374686ms', 'executionTime': '145.147882ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Create primary index
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default
2022-09-02 03:28:47 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 671.018664ms
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.create_primary_index] Check if index is online
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.488701ms
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-02 03:28:48 | INFO | MainProcess | MainThread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:28:49 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.996234ms
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.603157ms
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.537300159591324, 'mem_free': 13455237120, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:50 | INFO | MainProcess | MainThread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:51 | INFO | MainProcess | MainThread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:54 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 suite_setUp ==============
2022-09-02 03:28:54 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:28:55 | ERROR | MainProcess | MainThread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 suite_setUp ==============
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:28:55 | INFO | MainProcess | MainThread | [basetestcase.tearDown] closing all memcached connections
ok
----------------------------------------------------------------------
Ran 1 test in 148.913s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Cluster instance shutdown with force
-->result:
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.get_nodes_version] Node version in cluster 7.2.0-1948-rel-EE-enterprise
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index==============
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:56 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.428749156005401, 'mem_free': 13450723328, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:28:57 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:28:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:29:01 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:01 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
2022-09-02 03:29:02 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '151', 'memoryTotal': 15466930176, 'memoryFree': 13424906240, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 127.0.0.1
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7650
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
2022-09-02 03:29:03 | ERROR | MainProcess | Cluster_Thread | [on_prem_rest_client._http_request] POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9000
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '149', 'memoryTotal': 15466930176, 'memoryFree': 13435834368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9001
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:03 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '150', 'memoryTotal': 15466930176, 'memoryFree': 13435478016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9002
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
2022-09-02 03:29:04 | INFO | MainProcess | Cluster_Thread | [task.execute] {'uptime': '149', 'memoryTotal': 15466930176, 'memoryFree': 13426843648, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=7906
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> in init_cluster...Administrator,asdasd,9003
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.init_cluster] --> status:True
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
2022-09-02 03:29:05 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=memory_optimized
2022-09-02 03:29:05 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
2022-09-02 03:29:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ...
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:29:10 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [remote_util.enable_diag_eval_on_non_local_hosts] b'ok'
2022-09-02 03:29:11 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 127.0.0.1
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.create_bucket] 0.06 seconds to create bucket default
2022-09-02 03:29:11 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:05 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 7650
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:06 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.496182174896917, 'mem_free': 13424103424, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] Time to execute basesetup : 74.18479132652283
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 127.0.0.1 cluster is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 127.0.0.1 is healthy
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.settings.storage_mode': 'memory_optimized'} set
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Allowing the indexer to complete restart after setting the internal settings
2022-09-02 03:30:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ...
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings_internal] {'indexer.api.enableTestServer': True} set
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:15 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:16 | INFO | MainProcess | test_thread | [basetestcase.load] create 2016.0 to default documents...
2022-09-02 03:30:16 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 127.0.0.1:12000 default
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:127.0.0.1 port:9000 ssh_username:Administrator
2022-09-02 03:30:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. ...
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index existed in default on server 127.0.0.1
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 62.537338ms
2022-09-02 03:30:48 | ERROR | MainProcess | test_thread | [tuq_helper._is_index_in_list] Fail to get index list. List output: {'requestID': 'd4c3aeb8-8f2a-4a22-9902-74b7973d8d47', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.537338ms', 'executionTime': '62.476347ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Create primary index
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default
2022-09-02 03:30:48 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default+
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 645.309013ms
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] Check if index is online
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 6.361302ms
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'queryport.client.waitForScheduledIndex': False} set
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
2022-09-02 03:30:49 | INFO | MainProcess | test_thread | [on_prem_rest_client.set_index_settings] {'indexer.allowScheduleCreateRebal': True} set
2022-09-02 03:30:50 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX `employeecbb9fb710ec44faa9ea4295a8527c075job_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=CREATE+INDEX+%60employeecbb9fb710ec44faa9ea4295a8527c075job_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
2022-09-02 03:30:50 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 30.75446ms
2022-09-02 03:30:50 | INFO | MainProcess | test_thread | [base_gsi.async_build_index] BUILD INDEX on default(employeecbb9fb710ec44faa9ea4295a8527c075job_title) USING GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY BUILD INDEX on default(employeecbb9fb710ec44faa9ea4295a8527c075job_title) USING GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=BUILD+INDEX+on+default%28employeecbb9fb710ec44faa9ea4295a8527c075job_title%29+USING+GSI
2022-09-02 03:30:51 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 24.016115ms
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:53 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.288459ms
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:54 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 8.261519ms
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute] <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 2.444519ms
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute] {'requestID': 'eb8b0326-c7ef-40ba-b002-28ae7f8092df', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeecbb9fb710ec44faa9ea4295a8527c075job_title', 'index_id': 'e8ce974648532c66', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.444519ms', 'executionTime': '2.37641ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.execute] <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | Cluster_Thread | [task.check] <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [base_gsi.async_query_using_index] Query : SELECT * FROM default WHERE job_title = "Sales"
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] FROM clause ===== is default
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] WHERE clause ===== is doc["job_title"] == "Sales"
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] UNNEST clause ===== is None
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators.generate_expected_result] SELECT clause ===== is {"*" : doc,}
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
2022-09-02 03:30:55 | INFO | MainProcess | test_thread | [tuq_generators._filter_full_set] -->where_clause= doc["job_title"] == "Sales"
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [task.execute] <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 116.603404ms
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results] Analyzing Actual Result
2022-09-02 03:30:56 | INFO | MainProcess | Cluster_Thread | [tuq_helper._verify_results] Analyzing Expected Result
2022-09-02 03:30:57 | INFO | MainProcess | Cluster_Thread | [task.execute] <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:57 | INFO | MainProcess | Cluster_Thread | [task.check] <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.732816ms
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX employeecbb9fb710ec44faa9ea4295a8527c075job_title ON default USING GSI
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=DROP+INDEX+employeecbb9fb710ec44faa9ea4295a8527c075job_title+ON+default+USING+GSI
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 37.547272ms
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = 'employeecbb9fb710ec44faa9ea4295a8527c075job_title'
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeecbb9fb710ec44faa9ea4295a8527c075job_title%27
2022-09-02 03:30:58 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 5.384594ms
2022-09-02 03:30:58 | ERROR | MainProcess | Cluster_Thread | [tuq_helper._is_index_in_list] Fail to get index list. List output: {'requestID': 'ee8a75b4-8d06-40c1-8b2c-109f25bf9ff0', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.384594ms', 'executionTime': '5.327267ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:58 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.diag_eval] /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes where name = '#primary'
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 7.246185ms
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [on_prem_rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 49.795454ms
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics -------
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 4.707996668458969, 'mem_free': 13347934208, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics ---
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:30:59 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
2022-09-02 03:31:00 | INFO | MainProcess | test_thread | [remote_util.ssh_connect_with_retries] SSH Connected to 127.0.0.1 as Administrator
2022-09-02 03:31:01 | INFO | MainProcess | test_thread | [remote_util.extract_remote_info] extract_remote_info-->distribution_type: linux, distribution_version: default
2022-09-02 03:31:04 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:31:04 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets ['default'] on 127.0.0.1
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete....
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.bucket_exists] node 127.0.0.1 existing buckets : []
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 127.0.0.1
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] Could not find any buckets for node 127.0.0.1, nothing to delete
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] Removing user 'clientuser'...
2022-09-02 03:31:05 | ERROR | MainProcess | test_thread | [on_prem_rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] b'"User was not found."'
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9001
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9001 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9002
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9002 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9003
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.is_ns_server_running] -->is_ns_server_running?
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9003 is running
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ==============
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all ssh connections
2022-09-02 03:31:05 | INFO | MainProcess | test_thread | [basetestcase.tearDown] closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_1
ok
----------------------------------------------------------------------
Ran 1 test in 129.948s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,delete_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'delete_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 2, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2'}
[2022-09-02 03:31:05,996] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:05,997] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:06,261] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:06,292] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:31:06,373] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:31:06,374] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #2 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:31:06,375] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:31:07,026] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:31:07,055] - [task:164] INFO - {'uptime': '276', 'memoryTotal': 15466930176, 'memoryFree': 13315837952, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:07,083] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:31:07,083] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:31:07,084] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:31:07,117] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:31:07,155] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:31:07,156] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:31:07,183] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:31:07,184] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:31:07,185] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:31:07,185] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:31:07,234] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:07,237] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:07,238] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:07,492] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:07,493] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:07,559] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:07,560] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:07,591] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:07,592] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:07,714] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:31:07,740] - [task:164] INFO - {'uptime': '275', 'memoryTotal': 15466930176, 'memoryFree': 13346504704, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:07,765] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:07,792] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:31:07,792] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:31:07,843] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:07,847] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:07,848] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:08,100] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:08,101] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:08,169] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:08,170] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:08,202] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:08,203] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:08,320] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:31:08,347] - [task:164] INFO - {'uptime': '275', 'memoryTotal': 15466930176, 'memoryFree': 13346648064, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:08,372] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:08,400] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:31:08,400] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:31:08,452] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:08,455] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:08,455] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:08,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:08,709] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:08,775] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:08,776] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:08,805] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:08,806] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:08,923] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:31:08,950] - [task:164] INFO - {'uptime': '269', 'memoryTotal': 15466930176, 'memoryFree': 13346852864, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:31:08,977] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:31:09,005] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:31:09,005] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:31:09,057] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:31:09,060] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:09,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:09,314] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:09,315] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:09,381] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:09,382] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:09,411] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:31:09,412] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:31:09,501] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:31:09,884] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:31:14,888] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:31:14,973] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:31:14,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:31:14,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:31:15,238] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:31:15,239] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:31:15,306] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:31:15,307] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:31:15,307] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:31:15,613] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:31:15,669] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:31:15,670] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:32:05,579] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:05,941] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:06,239] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:32:06,241] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:32:06,300] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,300] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:06,574] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:06,579] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,579] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:06,825] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:06,830] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:06,830] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:07,076] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:07,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:07,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:07,420] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:11,128] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:32:11,128] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 7.215339608572898, 'mem_free': 13328629760, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:32:11,128] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:32:11,129] - [basetestcase:467] INFO - Time to execute basesetup : 65.13436388969421
[2022-09-02 03:32:11,180] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,180] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:32:11,233] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,233] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:32:11,287] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,287] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:32:11,347] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:32:11,347] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:32:11,403] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:11,464] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:32:11,465] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:32:11,465] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:32:16,472] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:32:16,475] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:32:16,475] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:32:16,732] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:32:17,719] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:32:17,883] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:20,543] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:32:20,558] - [newtuq:85] INFO - {'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:32:21,224] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:21,224] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:32:21,224] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:32:51,255] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:32:51,284] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:32:51,309] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:32:51,380] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.935505ms
[2022-09-02 03:32:51,380] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'b2ed11eb-34f3-4452-8bc6-a2d37ebb2339', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.935505ms', 'executionTime': '68.876283ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:32:51,380] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:32:51,407] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:32:51,432] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:32:52,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 680.564482ms
[2022-09-02 03:32:52,115] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:32:52,181] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:32:52,212] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:32:52,221] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.43582ms
[2022-09-02 03:32:52,455] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:52,491] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:32:52,504] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:32:52,506] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:32:52,517] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:32:52,576] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:32:53,400] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee2d203bfa69194995af9d55b941b297ccjob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:32:53,427] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee2d203bfa69194995af9d55b941b297ccjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:32:53,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.754369ms
[2022-09-02 03:32:53,459] - [base_gsi:282] INFO - BUILD INDEX on default(employee2d203bfa69194995af9d55b941b297ccjob_title) USING GSI
[2022-09-02 03:32:54,490] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee2d203bfa69194995af9d55b941b297ccjob_title) USING GSI
[2022-09-02 03:32:54,515] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee2d203bfa69194995af9d55b941b297ccjob_title%29+USING+GSI
[2022-09-02 03:32:54,536] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.133108ms
[2022-09-02 03:32:55,565] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:32:55,590] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:32:55,599] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.184288ms
[2022-09-02 03:32:56,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:32:56,654] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:32:56,663] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.368873ms
[2022-09-02 03:32:57,178] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-02 03:32:57,344] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:32:58,196] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:32:58,666] - [task:3235] INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:32:58,699] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:32:58,726] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:32:58,731] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.117003ms
[2022-09-02 03:32:58,731] - [task:3245] INFO - {'requestID': '169620b7-bebe-47e4-acb5-8ce8ac038719', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee2d203bfa69194995af9d55b941b297ccjob_title', 'index_id': 'e689fc1e0c806489', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.117003ms', 'executionTime': '3.039588ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:32:58,731] - [task:3246] INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:32:58,731] - [task:3276] INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:32:58,732] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:32:58,732] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:32:58,733] - [tuq_generators:72] INFO - WHERE clause ===== is doc["job_title"] == "Sales"
[2022-09-02 03:32:58,733] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:32:58,733] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:32:58,733] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:32:58,733] - [tuq_generators:329] INFO - -->where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:32:59,733] - [task:3235] INFO - <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:32:59,763] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:32:59,789] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:32:59,912] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 113.002601ms
[2022-09-02 03:32:59,912] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:32:59,912] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:33:00,723] - [task:3246] INFO - <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:33:00,723] - [task:3276] INFO - <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:33:01,755] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:33:01,782] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:33:01,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.101188ms
[2022-09-02 03:33:01,819] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee2d203bfa69194995af9d55b941b297ccjob_title ON default USING GSI
[2022-09-02 03:33:01,848] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee2d203bfa69194995af9d55b941b297ccjob_title+ON+default+USING+GSI
[2022-09-02 03:33:01,890] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.009202ms
[2022-09-02 03:33:01,928] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee2d203bfa69194995af9d55b941b297ccjob_title'
[2022-09-02 03:33:01,957] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee2d203bfa69194995af9d55b941b297ccjob_title%27
[2022-09-02 03:33:01,964] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.43114ms
[2022-09-02 03:33:01,964] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '11076b53-8242-464f-b40d-bb6e0441a38b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.43114ms', 'executionTime': '5.361373ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:33:02,077] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:02,081] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:02,081] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:02,462] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:02,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:33:02,519] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:33:02,585] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:33:02,643] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:02,646] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:02,646] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:03,024] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:03,082] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:33:03,082] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:33:03,143] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:33:03,202] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:33:03,202] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:33:03,202] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:33:03,229] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:33:03,255] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:33:03,261] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.28082ms
[2022-09-02 03:33:03,287] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:33:03,312] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:33:03,369] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.143543ms
[2022-09-02 03:33:03,430] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:33:03,430] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 20.72944656850717, 'mem_free': 13206835200, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:33:03,431] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:33:03,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:03,435] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:03,784] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:03,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:03,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:04,128] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:04,134] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:04,134] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:04,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:04,690] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:04,691] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:05,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:09,330] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:33:09,472] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:33:10,346] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:33:10,375] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:33:10,375] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:33:10,431] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,485] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,539] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:33:10,540] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:33:10,632] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:33:10,633] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:33:10,659] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,790] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:33:10,790] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,816] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:33:10,841] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:33:10,841] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,868] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:33:10,894] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:33:10,894] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,920] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:33:10,945] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:33:10,945] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:10,972] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:33:10,972] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #2 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:33:10,973] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:33:10,973] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_2
ok
----------------------------------------------------------------------
Ran 1 test in 125.030s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,update_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'update_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 3, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3'}
[2022-09-02 03:33:11,069] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:11,069] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:11,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:11,411] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:33:11,488] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:33:11,488] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #3 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:33:11,489] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:33:12,098] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:33:12,125] - [task:164] INFO - {'uptime': '401', 'memoryTotal': 15466930176, 'memoryFree': 13275095040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:12,151] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:33:12,151] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:33:12,152] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:33:12,183] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:33:12,222] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:33:12,223] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:33:12,251] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:33:12,252] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:33:12,300] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:12,303] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:12,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:12,608] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:12,609] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:12,684] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:12,686] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:12,715] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:12,716] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:12,842] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:33:12,868] - [task:164] INFO - {'uptime': '400', 'memoryTotal': 15466930176, 'memoryFree': 13218377728, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:12,895] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:12,923] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:33:12,923] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:33:12,974] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:12,977] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:12,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:13,281] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:13,282] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:13,357] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:13,358] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:13,391] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:13,391] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:13,504] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:33:13,530] - [task:164] INFO - {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13217734656, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:13,556] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:13,583] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:33:13,583] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:33:13,634] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:13,639] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:13,639] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:13,962] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:13,963] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:14,036] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:14,037] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:14,067] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:14,068] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:14,186] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:33:14,212] - [task:164] INFO - {'uptime': '395', 'memoryTotal': 15466930176, 'memoryFree': 13219360768, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:33:14,238] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:33:14,265] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:33:14,265] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:33:14,316] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:33:14,320] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:14,320] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:14,629] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:14,630] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:14,703] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:14,704] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:14,735] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:33:14,736] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:33:14,823] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:33:15,218] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:33:20,219] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:33:20,303] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:33:20,308] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:33:20,308] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:33:20,640] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:33:20,641] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:33:20,716] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:33:20,717] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:33:20,718] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:33:20,934] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:33:20,993] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:33:20,994] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:34:05,622] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:05,995] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:06,302] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:34:06,304] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:34:06,378] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:06,378] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:06,704] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:06,708] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:06,709] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,003] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:07,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:07,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,347] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:07,357] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:07,358] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:07,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:11,956] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:34:11,956] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.93267907264082, 'mem_free': 13270089728, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:34:11,956] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:34:11,956] - [basetestcase:467] INFO - Time to execute basesetup : 60.889695167541504
[2022-09-02 03:34:12,007] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,008] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:34:12,060] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,061] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:34:12,114] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,114] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:34:12,167] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:34:12,167] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:34:12,222] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:12,282] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:34:12,283] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:34:12,283] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:34:17,294] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:34:17,299] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:34:17,299] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:34:17,649] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:34:18,661] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:34:18,832] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:21,395] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:34:21,414] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:34:22,258] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:22,259] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:34:22,259] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:34:52,288] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:34:52,317] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:34:52,342] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:34:52,409] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 64.558727ms
[2022-09-02 03:34:52,409] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'af56e504-e83d-4978-bf7e-cbe460c0aa34', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '64.558727ms', 'executionTime': '64.442284ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:34:52,409] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:34:52,435] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:34:52,461] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:34:53,135] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 671.889114ms
[2022-09-02 03:34:53,135] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:34:53,192] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:34:53,234] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:34:53,241] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.805583ms
[2022-09-02 03:34:53,443] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:53,481] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:34:53,495] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:34:53,496] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:34:53,514] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:34:53,585] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:34:54,407] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee02be61b6900f45fd81049c85068a0a9ajob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:34:54,432] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee02be61b6900f45fd81049c85068a0a9ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:34:54,465] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 30.431676ms
[2022-09-02 03:34:54,466] - [base_gsi:282] INFO - BUILD INDEX on default(employee02be61b6900f45fd81049c85068a0a9ajob_title) USING GSI
[2022-09-02 03:34:55,501] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee02be61b6900f45fd81049c85068a0a9ajob_title) USING GSI
[2022-09-02 03:34:55,526] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee02be61b6900f45fd81049c85068a0a9ajob_title%29+USING+GSI
[2022-09-02 03:34:55,548] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 19.614095ms
[2022-09-02 03:34:56,580] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:34:56,608] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:34:56,618] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.068854ms
[2022-09-02 03:34:57,648] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:34:57,673] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:34:57,680] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.566114ms
[2022-09-02 03:34:58,155] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:34:58,323] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:34:59,023] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:34:59,683] - [task:3235] INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:34:59,716] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:34:59,744] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:34:59,748] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.280499ms
[2022-09-02 03:34:59,748] - [task:3245] INFO - {'requestID': '84496e26-bef4-4720-9d43-a5b9cd9e93cc', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee02be61b6900f45fd81049c85068a0a9ajob_title', 'index_id': 'ab2090ba08dbba3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '2.280499ms', 'executionTime': '2.20169ms', 'resultCount': 1, 'resultSize': 723, 'serviceLoad': 6}}
[2022-09-02 03:34:59,749] - [task:3246] INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:34:59,749] - [task:3276] INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:34:59,749] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:34:59,750] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:34:59,750] - [tuq_generators:72] INFO - WHERE clause ===== is doc["job_title"] == "Sales"
[2022-09-02 03:34:59,750] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:34:59,751] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:34:59,751] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:34:59,751] - [tuq_generators:329] INFO - -->where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:35:00,751] - [task:3235] INFO - <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:35:00,781] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:35:00,808] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:35:00,907] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 89.700283ms
[2022-09-02 03:35:00,907] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:35:00,908] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:35:01,714] - [task:3246] INFO - <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:35:01,714] - [task:3276] INFO - <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:35:02,748] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:35:02,777] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:35:02,785] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.468333ms
[2022-09-02 03:35:02,814] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee02be61b6900f45fd81049c85068a0a9ajob_title ON default USING GSI
[2022-09-02 03:35:02,840] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee02be61b6900f45fd81049c85068a0a9ajob_title+ON+default+USING+GSI
[2022-09-02 03:35:02,896] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 54.02186ms
[2022-09-02 03:35:02,931] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee02be61b6900f45fd81049c85068a0a9ajob_title'
[2022-09-02 03:35:02,959] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee02be61b6900f45fd81049c85068a0a9ajob_title%27
[2022-09-02 03:35:02,966] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.603696ms
[2022-09-02 03:35:02,967] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '26f2d73b-fead-46fa-8d65-ccbdacb07507', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.603696ms', 'executionTime': '5.531127ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:35:03,082] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:03,086] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:03,086] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:03,471] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:03,527] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:35:03,527] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:35:03,596] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:35:03,651] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:03,654] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:03,654] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:04,046] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:04,110] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:35:04,110] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:35:04,183] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:35:04,241] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:35:04,241] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:35:04,242] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:35:04,269] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:35:04,296] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:35:04,303] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.598169ms
[2022-09-02 03:35:04,329] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:35:04,355] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:35:04,405] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 48.501231ms
[2022-09-02 03:35:04,474] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:35:04,474] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 18.47086309230359, 'mem_free': 13138907136, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:35:04,474] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:35:04,479] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:04,479] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:04,874] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:04,879] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:04,879] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:05,473] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:05,480] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:05,481] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:06,197] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:06,205] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:06,205] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:06,917] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:11,166] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:35:11,318] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:35:12,270] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:35:12,298] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:35:12,298] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:35:12,353] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,408] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,476] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:35:12,477] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:35:12,559] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:35:12,560] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:35:12,587] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,719] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:35:12,719] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,749] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:35:12,777] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:35:12,777] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,804] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:35:12,830] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:35:12,831] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,858] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:35:12,885] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:35:12,885] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:12,914] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:35:12,914] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #3 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:35:12,914] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:35:12,915] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_3
ok
----------------------------------------------------------------------
Ran 1 test in 121.900s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,expiry_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'expiry_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 4, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4'}
[2022-09-02 03:35:13,017] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:13,017] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:13,394] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:13,424] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:35:13,504] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:35:13,505] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #4 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:35:13,505] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:35:14,047] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:35:14,075] - [task:164] INFO - {'uptime': '523', 'memoryTotal': 15466930176, 'memoryFree': 13237760000, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:14,103] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:35:14,103] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:35:14,104] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:35:14,145] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:35:14,183] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:35:14,184] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:35:14,215] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:35:14,216] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:35:14,268] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:14,273] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:14,273] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:14,675] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:14,676] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:14,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:14,763] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:14,793] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:14,794] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:14,918] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:35:14,947] - [task:164] INFO - {'uptime': '521', 'memoryTotal': 15466930176, 'memoryFree': 13132648448, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:14,974] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:15,003] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:35:15,003] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:35:15,056] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:15,059] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:15,060] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:15,446] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:15,447] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:15,533] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:15,534] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:15,568] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:15,568] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:15,685] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:35:15,711] - [task:164] INFO - {'uptime': '521', 'memoryTotal': 15466930176, 'memoryFree': 13231337472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:15,737] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:15,765] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:35:15,766] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:35:15,819] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:15,824] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:15,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:16,219] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:16,220] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:16,314] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:16,315] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:16,347] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:16,348] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:16,477] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:35:16,507] - [task:164] INFO - {'uptime': '520', 'memoryTotal': 15466930176, 'memoryFree': 13246390272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:35:16,535] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:35:16,564] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:35:16,564] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:35:16,618] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:35:16,621] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:16,622] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:17,037] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:17,038] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:17,131] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:17,132] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:17,165] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:35:17,166] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:35:17,256] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:35:17,674] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:35:22,676] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:35:22,766] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:35:22,772] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:35:22,772] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:35:23,188] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:35:23,189] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:35:23,277] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:35:23,278] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:35:23,279] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:35:24,372] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:35:24,429] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:35:24,430] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:36:05,406] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:05,722] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:06,239] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:36:06,242] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:36:06,303] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:06,304] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:06,689] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:06,694] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:06,694] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:07,048] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:07,055] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:07,055] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:07,591] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:07,598] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:07,598] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:08,253] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:12,702] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:36:12,702] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.72395422371751, 'mem_free': 13239820288, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:36:12,702] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:36:12,703] - [basetestcase:467] INFO - Time to execute basesetup : 59.6883327960968
[2022-09-02 03:36:12,758] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,758] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:36:12,810] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,810] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:36:12,863] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,864] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:36:12,918] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:36:12,918] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:36:12,969] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:13,035] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:36:13,035] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:36:13,035] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:36:18,043] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:36:18,046] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:36:18,046] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:36:18,414] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:36:19,438] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:36:19,607] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:22,127] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:36:22,150] - [newtuq:85] INFO - {'expiry': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}}
[2022-09-02 03:36:22,833] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:22,833] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:36:22,833] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:36:52,862] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:36:52,896] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:36:52,925] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:36:52,997] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 71.047189ms
[2022-09-02 03:36:52,998] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '0c341982-0299-4f21-9e3e-5ba4b32756a4', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '71.047189ms', 'executionTime': '70.980201ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:36:52,998] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:36:53,027] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:36:53,054] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:36:53,737] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.542504ms
[2022-09-02 03:36:53,738] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:36:53,816] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:36:53,851] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:36:53,859] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.031534ms
[2022-09-02 03:36:54,080] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:54,118] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:36:54,136] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:36:54,139] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:36:54,174] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:36:54,236] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:36:55,037] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee815de908b3294035881b89a82c2cb77fjob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:36:55,065] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee815de908b3294035881b89a82c2cb77fjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:36:55,106] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.8135ms
[2022-09-02 03:36:55,107] - [base_gsi:282] INFO - BUILD INDEX on default(employee815de908b3294035881b89a82c2cb77fjob_title) USING GSI
[2022-09-02 03:36:56,138] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee815de908b3294035881b89a82c2cb77fjob_title) USING GSI
[2022-09-02 03:36:56,163] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee815de908b3294035881b89a82c2cb77fjob_title%29+USING+GSI
[2022-09-02 03:36:56,191] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 26.952335ms
[2022-09-02 03:36:57,223] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:36:57,248] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:36:57,257] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.04057ms
[2022-09-02 03:36:58,288] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:36:58,316] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:36:58,325] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.737056ms
[2022-09-02 03:36:58,792] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:36:58,967] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:36:59,773] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:36:59,972] - [data_helper:309] INFO - dict:{'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}
[2022-09-02 03:36:59,972] - [data_helper:310] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:37:00,153] - [cluster_helper:379] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10 on default
[2022-09-02 03:37:00,154] - [mc_bin_client:669] INFO - setting param: exp_pager_stime 10
[2022-09-02 03:37:00,154] - [cluster_helper:393] INFO - Setting flush param on server {'ip': '127.0.0.1', 'port': '9000', 'username': 'Administrator', 'password': 'asdasd'}, exp_pager_stime to 10, result: (2086350291, 0, b'')
[2022-09-02 03:37:00,327] - [task:3235] INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:00,356] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:37:00,384] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:37:00,389] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.002272ms
[2022-09-02 03:37:00,389] - [task:3245] INFO - {'requestID': '1b29e3a3-1887-4aa1-b689-3652399e56fd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee815de908b3294035881b89a82c2cb77fjob_title', 'index_id': '7c3fa2286002c776', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.002272ms', 'executionTime': '2.941184ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:37:00,389] - [task:3246] INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:00,389] - [task:3276] INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:00,390] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:37:00,390] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:37:00,390] - [tuq_generators:72] INFO - WHERE clause ===== is doc["job_title"] == "Sales"
[2022-09-02 03:37:00,391] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:37:00,391] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:37:00,391] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:37:00,391] - [tuq_generators:329] INFO - -->where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:37:01,391] - [task:3235] INFO - <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:01,421] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:37:01,449] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:37:01,570] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 112.537957ms
[2022-09-02 03:37:01,570] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:37:01,571] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:37:02,400] - [task:3246] INFO - <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:02,400] - [task:3276] INFO - <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:37:03,432] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:37:03,459] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:37:03,467] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.490259ms
[2022-09-02 03:37:03,495] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee815de908b3294035881b89a82c2cb77fjob_title ON default USING GSI
[2022-09-02 03:37:03,524] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee815de908b3294035881b89a82c2cb77fjob_title+ON+default+USING+GSI
[2022-09-02 03:37:03,563] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.321182ms
[2022-09-02 03:37:03,600] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee815de908b3294035881b89a82c2cb77fjob_title'
[2022-09-02 03:37:03,628] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee815de908b3294035881b89a82c2cb77fjob_title%27
[2022-09-02 03:37:03,635] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.210414ms
[2022-09-02 03:37:03,635] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '04d1b611-644d-41e8-a8b8-0a786b850b8d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.210414ms', 'executionTime': '5.148652ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:37:03,741] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:03,745] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:03,745] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:04,246] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:04,302] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:37:04,303] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:37:04,379] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:37:04,437] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:04,440] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:04,441] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:04,949] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:05,017] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:37:05,017] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:37:05,096] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:37:05,156] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:37:05,157] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:37:05,157] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:37:05,185] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:37:05,215] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:37:05,222] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.43078ms
[2022-09-02 03:37:05,251] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:37:05,279] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:37:05,338] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 56.490743ms
[2022-09-02 03:37:05,409] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:37:05,410] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 19.38479004483664, 'mem_free': 13098631168, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:37:05,410] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:37:05,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:05,415] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:05,904] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:05,909] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:05,909] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:06,396] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:06,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:06,402] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:06,866] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:06,871] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:06,871] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:07,680] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:13,396] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:37:13,540] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:37:14,305] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:37:14,334] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:37:14,334] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:37:14,389] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,442] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,496] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:37:14,497] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:37:14,576] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:37:14,578] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:37:14,605] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,754] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:37:14,754] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,780] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:37:14,806] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:37:14,806] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,833] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:37:14,858] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:37:14,859] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,886] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:37:14,911] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:37:14,911] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:14,938] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:37:14,938] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #4 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:37:14,939] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:37:14,939] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 4 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_4
ok
----------------------------------------------------------------------
Ran 1 test in 121.977s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,run_async=True,scan_consistency=request_plus,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 5, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5'}
[2022-09-02 03:37:15,038] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:15,038] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:15,457] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:15,488] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:37:15,570] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:37:15,570] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #5 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:37:15,571] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:37:16,068] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:37:16,099] - [task:164] INFO - {'uptime': '645', 'memoryTotal': 15466930176, 'memoryFree': 13202739200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:16,124] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:37:16,124] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:37:16,125] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:37:16,157] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:37:16,195] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:37:16,195] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:37:16,223] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:37:16,224] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:37:16,224] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:37:16,225] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:37:16,273] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:16,276] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:16,276] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:16,708] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:16,709] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:16,807] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:16,808] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:16,840] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:16,841] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:16,973] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:37:17,002] - [task:164] INFO - {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13124141056, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:17,028] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:17,055] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:37:17,056] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:37:17,109] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:17,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:17,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:17,600] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:17,601] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:17,703] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:17,704] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:17,737] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:17,737] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:17,860] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:37:17,890] - [task:164] INFO - {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13202726912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:17,917] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:17,946] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:37:17,947] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:37:18,002] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:18,007] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:18,007] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:18,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:18,481] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:18,575] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:18,576] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:18,608] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:18,609] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:18,724] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:37:18,751] - [task:164] INFO - {'uptime': '641', 'memoryTotal': 15466930176, 'memoryFree': 13202874368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:37:18,777] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:37:18,804] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:37:18,804] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:37:18,856] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:37:18,860] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:18,860] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:19,332] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:19,333] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:19,431] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:19,432] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:19,461] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:37:19,461] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:37:19,554] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:37:20,003] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:37:25,006] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:37:25,095] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:37:25,100] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:37:25,100] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:37:25,548] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:37:25,550] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:37:25,641] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:37:25,642] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:37:25,642] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:37:26,707] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:37:26,762] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:37:26,763] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:38:05,692] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:06,014] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:06,297] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:38:06,300] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:38:06,362] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:06,362] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:06,785] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:06,790] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:06,791] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:07,210] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:07,215] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:07,215] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:07,940] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:07,947] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:07,947] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:08,683] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:13,815] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:38:13,816] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.94294453438017, 'mem_free': 13204148224, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:38:13,816] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:38:13,816] - [basetestcase:467] INFO - Time to execute basesetup : 58.7807400226593
[2022-09-02 03:38:13,867] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,867] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:38:13,920] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,921] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:38:13,977] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:13,977] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:38:14,036] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:38:14,036] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:38:14,086] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:14,144] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:38:14,144] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:38:14,144] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:38:19,153] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:38:19,156] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:38:19,157] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:38:19,585] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:38:20,670] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:38:20,839] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:38:23,256] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:38:23,272] - [newtuq:85] INFO - {'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-02 03:38:24,348] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:24,348] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:38:24,348] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:38:54,377] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:38:54,406] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:38:54,432] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:38:54,497] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.149085ms
[2022-09-02 03:38:54,497] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '7e8e72ad-145b-4767-8337-02d1bba709a7', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.149085ms', 'executionTime': '62.083293ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:38:54,497] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:38:54,523] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:38:54,550] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:38:55,263] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 711.546631ms
[2022-09-02 03:38:55,264] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:38:55,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:38:55,365] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:38:55,373] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.168973ms
[2022-09-02 03:38:55,582] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:55,618] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:38:55,633] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:38:55,634] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:38:55,649] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:38:55,721] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:38:56,545] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee7c05dfae775b4e799ad9a6cf48fc842ajob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:38:56,571] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:38:56,604] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.416821ms
[2022-09-02 03:38:56,605] - [base_gsi:282] INFO - BUILD INDEX on default(employee7c05dfae775b4e799ad9a6cf48fc842ajob_title) USING GSI
[2022-09-02 03:38:57,635] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee7c05dfae775b4e799ad9a6cf48fc842ajob_title) USING GSI
[2022-09-02 03:38:57,660] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%29+USING+GSI
[2022-09-02 03:38:57,685] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.860941ms
[2022-09-02 03:38:58,717] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:38:58,744] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:38:58,754] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.341981ms
[2022-09-02 03:38:59,787] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:38:59,816] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:38:59,822] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.062601ms
[2022-09-02 03:39:00,310] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:39:00,476] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:39:04,078] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:39:04,828] - [task:3235] INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:04,863] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:39:04,889] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:39:04,895] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.030947ms
[2022-09-02 03:39:04,895] - [task:3245] INFO - {'requestID': '784be84b-e467-40dd-b7e8-bd4cfea9e8df', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title', 'index_id': '33279191c77201c6', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.030947ms', 'executionTime': '2.962671ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:39:04,895] - [task:3246] INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:04,896] - [task:3276] INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:04,896] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:39:04,897] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:39:04,897] - [tuq_generators:72] INFO - WHERE clause ===== is doc["job_title"] == "Sales"
[2022-09-02 03:39:04,897] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:39:04,898] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:39:04,898] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:39:04,898] - [tuq_generators:329] INFO - -->where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:39:05,897] - [task:3235] INFO - <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:05,930] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:39:05,956] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:39:06,115] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 145.034406ms
[2022-09-02 03:39:06,115] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:39:06,116] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:39:07,729] - [task:3246] INFO - <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:07,729] - [task:3276] INFO - <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:39:08,760] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:39:08,785] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:39:08,792] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.679802ms
[2022-09-02 03:39:08,819] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee7c05dfae775b4e799ad9a6cf48fc842ajob_title ON default USING GSI
[2022-09-02 03:39:08,844] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee7c05dfae775b4e799ad9a6cf48fc842ajob_title+ON+default+USING+GSI
[2022-09-02 03:39:08,886] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.153314ms
[2022-09-02 03:39:08,922] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee7c05dfae775b4e799ad9a6cf48fc842ajob_title'
[2022-09-02 03:39:08,947] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee7c05dfae775b4e799ad9a6cf48fc842ajob_title%27
[2022-09-02 03:39:08,955] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.345381ms
[2022-09-02 03:39:08,955] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '1e9259d3-86aa-4fa1-945c-cc51fc2161bb', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.345381ms', 'executionTime': '5.271523ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:39:09,059] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:09,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:09,062] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:09,575] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:09,631] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:39:09,631] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:39:09,722] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:39:09,778] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:09,781] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:09,781] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:10,319] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:10,377] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:39:10,377] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:39:10,470] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:39:10,526] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:39:10,526] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:39:10,526] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:39:10,551] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:39:10,577] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:39:10,584] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.794432ms
[2022-09-02 03:39:10,610] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:39:10,636] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:39:10,684] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.93437ms
[2022-09-02 03:39:10,759] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:39:10,760] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 15.14824495541436, 'mem_free': 13057966080, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:39:10,760] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:39:10,764] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:10,764] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:11,328] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:11,333] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:11,333] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:11,878] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:11,883] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:11,883] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:12,694] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:12,702] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:12,702] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:13,631] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:19,758] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:39:19,901] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:39:20,297] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:39:20,325] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:39:20,325] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:39:20,379] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,433] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,488] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:39:20,489] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:39:20,570] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:39:20,571] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:39:20,598] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,732] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:39:20,733] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,759] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:39:20,785] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:39:20,785] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,812] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:39:20,837] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:39:20,838] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,864] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:39:20,890] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:39:20,890] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:20,917] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:39:20,917] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #5 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:39:20,917] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:39:20,918] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 5 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_5
ok
----------------------------------------------------------------------
Ran 1 test in 125.935s
OK
test_multi_create_query_explain_drop_index (gsi.indexscans_gsi.SecondaryIndexingScanTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,doc_ops=True,create_ops_per=.5,delete_ops_per=.2,update_ops_per=.2,run_async=True,scan_consistency=request_plus,GROUP=gsi
Test Input params:
{'groups': 'simple:equals:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'doc_ops': 'True', 'create_ops_per': '.5', 'delete_ops_per': '.2', 'update_ops_per': '.2', 'run_async': 'True', 'scan_consistency': 'request_plus', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 6, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6'}
[2022-09-02 03:39:21,034] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:21,035] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:21,525] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:21,557] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:39:21,641] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:39:21,641] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #6 test_multi_create_query_explain_drop_index==============
[2022-09-02 03:39:21,642] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:39:22,058] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:39:22,086] - [task:164] INFO - {'uptime': '771', 'memoryTotal': 15466930176, 'memoryFree': 13143281664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:22,112] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:39:22,112] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:39:22,113] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:39:22,146] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:39:22,180] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:39:22,180] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:39:22,210] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:39:22,211] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:39:22,212] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:39:22,212] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:39:22,260] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:22,263] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:22,263] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:22,768] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:22,769] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:22,873] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:22,874] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:22,908] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:22,909] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:23,038] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:39:23,064] - [task:164] INFO - {'uptime': '772', 'memoryTotal': 15466930176, 'memoryFree': 13145874432, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:23,090] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:23,116] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:39:23,117] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:39:23,168] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:23,171] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:23,171] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:23,660] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:23,661] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:23,762] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:23,764] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:23,794] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:23,794] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:23,910] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:39:23,936] - [task:164] INFO - {'uptime': '767', 'memoryTotal': 15466930176, 'memoryFree': 13146128384, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:23,966] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:23,993] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:39:23,994] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:39:24,045] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:24,050] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:24,050] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:24,550] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:24,551] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:24,651] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:24,652] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:24,682] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:24,682] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:24,800] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:39:24,828] - [task:164] INFO - {'uptime': '767', 'memoryTotal': 15466930176, 'memoryFree': 13144543232, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:39:24,854] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:39:24,882] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:39:24,883] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:39:24,934] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:39:24,936] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:24,937] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:25,455] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:25,456] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:25,557] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:25,558] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:25,587] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:39:25,587] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:39:25,681] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:39:26,087] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:39:31,090] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:39:31,173] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:39:31,178] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:39:31,179] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:39:31,719] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:39:31,720] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:39:31,831] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:39:31,832] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:39:31,833] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:39:32,814] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:39:32,872] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:39:32,873] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:40:05,343] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:05,737] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:06,255] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:40:06,259] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:40:06,317] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:06,317] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:06,891] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:06,896] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:06,896] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:07,588] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:07,600] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:07,601] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:08,495] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:08,503] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:08,503] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:09,422] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:14,999] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:40:15,000] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.67130366951604, 'mem_free': 13144801280, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:40:15,000] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:40:15,000] - [basetestcase:467] INFO - Time to execute basesetup : 53.97261834144592
[2022-09-02 03:40:15,055] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,055] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:40:15,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,109] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:40:15,161] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,161] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:40:15,215] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:40:15,215] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:40:15,266] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:15,326] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:40:15,327] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:40:15,327] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:40:20,333] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:40:20,336] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:40:20,336] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:40:20,855] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:40:21,881] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:40:22,150] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:40:24,524] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:40:24,544] - [newtuq:85] INFO - {'update': {'start': 0, 'end': 0}, 'delete': {'start': 0, 'end': 0}, 'remaining': {'start': 0, 'end': 1}, 'create': {'start': 1, 'end': 2}}
[2022-09-02 03:40:25,935] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:25,936] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:40:25,936] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:40:55,951] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:40:55,980] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:40:56,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:40:56,070] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 62.970307ms
[2022-09-02 03:40:56,070] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'd4587a4b-6c65-4257-9a2d-d66ce12fcde2', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '62.970307ms', 'executionTime': '62.903828ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:40:56,070] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:40:56,096] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:40:56,120] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:40:56,791] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 668.841894ms
[2022-09-02 03:40:56,791] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:40:56,845] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:40:56,883] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:40:56,891] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.392064ms
[2022-09-02 03:40:57,172] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:57,238] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:40:57,256] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:40:57,257] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:40:57,274] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:40:57,344] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:40:58,081] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeeaead9de0b30c4b6f85b3617c98c7c16djob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:40:58,107] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:40:58,142] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.745217ms
[2022-09-02 03:40:58,143] - [base_gsi:282] INFO - BUILD INDEX on default(employeeaead9de0b30c4b6f85b3617c98c7c16djob_title) USING GSI
[2022-09-02 03:40:59,174] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeeaead9de0b30c4b6f85b3617c98c7c16djob_title) USING GSI
[2022-09-02 03:40:59,200] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%29+USING+GSI
[2022-09-02 03:40:59,226] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 24.324009ms
[2022-09-02 03:41:00,258] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:00,285] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:00,293] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.916549ms
[2022-09-02 03:41:01,325] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:01,352] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:01,360] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.828322ms
[2022-09-02 03:41:01,875] - [basetestcase:2772] INFO - update 0.0 to default documents...
[2022-09-02 03:41:02,054] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:03,151] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:03,835] - [basetestcase:2772] INFO - delete 0.0 to default documents...
[2022-09-02 03:41:04,043] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:05,203] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:05,598] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:41:05,776] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:41:09,512] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:41:10,382] - [task:3235] INFO - <<<<< START Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:10,414] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:41:10,441] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+
[2022-09-02 03:41:10,446] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.041758ms
[2022-09-02 03:41:10,446] - [task:3245] INFO - {'requestID': '0f630634-2747-4422-9b45-e09de310c682', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title', 'index_id': '99b553725d16e643', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '"Sales"', 'inclusion': 3, 'index_key': '`job_title`', 'low': '"Sales"'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((`default`.`job_title`) = "Sales")'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, 'text': 'SELECT * FROM default WHERE job_title = "Sales"'}], 'status': 'success', 'metrics': {'elapsedTime': '3.041758ms', 'executionTime': '2.974578ms', 'resultCount': 1, 'resultSize': 724, 'serviceLoad': 6}}
[2022-09-02 03:41:10,446] - [task:3246] INFO - <<<<< Done Executing Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:10,447] - [task:3276] INFO - <<<<< Done VERIFYING Query EXPLAIN SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:10,447] - [base_gsi:560] INFO - Query : SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:41:10,447] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:41:10,448] - [tuq_generators:72] INFO - WHERE clause ===== is doc["job_title"] == "Sales"
[2022-09-02 03:41:10,448] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:41:10,448] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:41:10,448] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,}; where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:41:10,449] - [tuq_generators:329] INFO - -->where_clause= doc["job_title"] == "Sales"
[2022-09-02 03:41:11,448] - [task:3235] INFO - <<<<< START Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:11,478] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE job_title = "Sales"
[2022-09-02 03:41:11,504] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++job_title+%3D+%22Sales%22+&scan_consistency=request_plus
[2022-09-02 03:41:11,660] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 141.911199ms
[2022-09-02 03:41:11,661] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:41:11,661] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:41:13,301] - [task:3246] INFO - <<<<< Done Executing Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:13,301] - [task:3276] INFO - <<<<< Done VERIFYING Query SELECT * FROM default WHERE job_title = "Sales" >>>>>>
[2022-09-02 03:41:14,331] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:14,356] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:14,363] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.746245ms
[2022-09-02 03:41:14,389] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeeaead9de0b30c4b6f85b3617c98c7c16djob_title ON default USING GSI
[2022-09-02 03:41:14,415] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeeaead9de0b30c4b6f85b3617c98c7c16djob_title+ON+default+USING+GSI
[2022-09-02 03:41:14,461] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 44.590207ms
[2022-09-02 03:41:14,494] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeeaead9de0b30c4b6f85b3617c98c7c16djob_title'
[2022-09-02 03:41:14,520] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeeaead9de0b30c4b6f85b3617c98c7c16djob_title%27
[2022-09-02 03:41:14,527] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.968596ms
[2022-09-02 03:41:14,527] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'ade83831-a589-4511-b20c-92439a99d474', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.968596ms', 'executionTime': '5.896591ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:41:14,631] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:14,634] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:14,635] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:15,284] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:15,339] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:41:15,339] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:41:15,444] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:41:15,503] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:15,506] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:15,506] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:16,165] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:16,222] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:41:16,222] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:41:16,327] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:41:16,382] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:41:16,383] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:41:16,383] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:41:16,408] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:41:16,433] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:41:16,441] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.161441ms
[2022-09-02 03:41:16,467] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:41:16,493] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:41:16,545] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 49.597112ms
[2022-09-02 03:41:16,612] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:41:16,612] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 25.75264437955159, 'mem_free': 12850675712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:41:16,613] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:41:16,616] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:16,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:17,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:17,380] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:17,380] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:17,998] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:18,003] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:18,003] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:18,804] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:18,812] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:18,812] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:19,928] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:27,736] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:41:27,878] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:41:28,380] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:41:28,410] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:41:28,411] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:41:28,470] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,525] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,578] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:41:28,579] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:41:28,656] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:41:28,657] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:41:28,682] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,810] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:41:28,810] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,837] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:41:28,863] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:41:28,863] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,890] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:41:28,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:41:28,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:28,942] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:41:28,968] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:41:28,969] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:29,000] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:41:29,001] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #6 test_multi_create_query_explain_drop_index ==============
[2022-09-02 03:41:29,001] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:41:29,002] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_6
ok
----------------------------------------------------------------------
Ran 1 test in 128.032s
OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=simple,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'simple', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 7, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7'}
[2022-09-02 03:41:29,145] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:29,146] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:29,728] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:29,761] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:41:29,841] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:41:29,841] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #7 test_multi_create_drop_index==============
[2022-09-02 03:41:29,841] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:41:30,172] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:41:30,202] - [task:164] INFO - {'uptime': '899', 'memoryTotal': 15466930176, 'memoryFree': 13087059968, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:30,228] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:41:30,229] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:41:30,229] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:41:30,259] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:41:30,295] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:41:30,295] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:41:30,327] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:41:30,328] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:41:30,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:30,383] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:30,383] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:30,986] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:30,987] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:31,100] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:31,101] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:31,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:31,133] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:31,268] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:41:31,295] - [task:164] INFO - {'uptime': '897', 'memoryTotal': 15466930176, 'memoryFree': 13087145984, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:31,321] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:31,348] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:41:31,349] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:41:31,401] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:31,404] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:31,405] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:31,996] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:31,997] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:32,111] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:32,112] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:32,142] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:32,143] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:32,263] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:41:32,290] - [task:164] INFO - {'uptime': '898', 'memoryTotal': 15466930176, 'memoryFree': 13086478336, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:32,315] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:32,343] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:41:32,343] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:41:32,397] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:32,402] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:32,403] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:33,021] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:33,022] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:33,137] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:33,139] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:33,169] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:33,170] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:33,287] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:41:33,313] - [task:164] INFO - {'uptime': '897', 'memoryTotal': 15466930176, 'memoryFree': 13086650368, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:41:33,339] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:41:33,366] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:41:33,367] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:41:33,425] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:41:33,428] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:33,428] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:34,027] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:34,028] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:34,143] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:34,144] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:34,173] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:41:34,174] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:41:34,267] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:41:34,695] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:41:39,700] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:41:39,783] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:41:39,797] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:41:39,797] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:41:40,392] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:41:40,393] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:41:40,517] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:41:40,518] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:41:40,518] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:41:41,401] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:41:41,454] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:41:41,455] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:42:05,763] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:06,074] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:06,333] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:42:06,336] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:42:06,390] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:06,390] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:06,971] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:06,976] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:06,977] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:07,762] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:07,775] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:07,775] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:08,811] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:08,821] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:08,821] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:09,868] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:16,175] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:42:16,176] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.80845304426461, 'mem_free': 13098573824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:42:16,176] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:42:16,176] - [basetestcase:467] INFO - Time to execute basesetup : 47.03368258476257
[2022-09-02 03:42:16,229] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,229] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:42:16,281] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,282] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:42:16,336] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,337] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:42:16,389] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:42:16,389] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:42:16,441] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:16,501] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:42:16,501] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:42:16,502] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:42:21,513] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:42:21,516] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:42:21,516] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:42:22,092] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:42:23,237] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:42:23,405] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:42:25,590] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:42:25,670] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:25,670] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:42:25,670] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:42:55,693] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:42:55,724] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:42:55,749] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:42:55,815] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.386652ms
[2022-09-02 03:42:55,815] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '9d101b02-3d1b-42c7-bb1b-d0cc107937c3', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.386652ms', 'executionTime': '63.326441ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:42:55,815] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:42:55,841] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:42:55,866] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:42:56,513] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 644.552306ms
[2022-09-02 03:42:56,513] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:42:56,572] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:42:56,614] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:42:56,623] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.370373ms
[2022-09-02 03:42:56,817] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:42:56,855] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:42:56,872] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:42:56,873] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:42:56,889] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:42:56,890] - [base_gsi:326] INFO - []
[2022-09-02 03:42:57,781] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title` ON default(job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:42:57,807] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%60+ON+default%28job_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:42:57,842] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 33.201279ms
[2022-09-02 03:42:57,876] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr` ON default(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'defer_build': True}
[2022-09-02 03:42:57,901] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:42:57,942] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 38.638628ms
[2022-09-02 03:42:57,944] - [base_gsi:282] INFO - BUILD INDEX on default(employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title,employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr) USING GSI
[2022-09-02 03:42:58,976] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title,employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr) USING GSI
[2022-09-02 03:42:59,001] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%2Cemployeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%29+USING+GSI
[2022-09-02 03:42:59,036] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 32.480995ms
[2022-09-02 03:43:00,067] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:00,094] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:00,103] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.437035ms
[2022-09-02 03:43:01,136] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:01,164] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:01,173] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.364551ms
[2022-09-02 03:43:01,200] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:01,226] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:01,230] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.763417ms
[2022-09-02 03:43:02,260] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:02,286] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:02,296] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.65221ms
[2022-09-02 03:43:02,321] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title ON default USING GSI
[2022-09-02 03:43:02,346] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title+ON+default+USING+GSI
[2022-09-02 03:43:02,401] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 53.162018ms
[2022-09-02 03:43:02,435] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:02,461] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:02,469] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.367097ms
[2022-09-02 03:43:02,495] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr ON default USING GSI
[2022-09-02 03:43:02,524] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr+ON+default+USING+GSI
[2022-09-02 03:43:02,562] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 36.884846ms
[2022-09-02 03:43:02,598] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title'
[2022-09-02 03:43:02,623] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjob_title%27
[2022-09-02 03:43:02,630] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.097315ms
[2022-09-02 03:43:02,630] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '12a42918-556b-4784-b452-db67c4cb01e8', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.097315ms', 'executionTime': '5.029226ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:43:02,655] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr'
[2022-09-02 03:43:02,680] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeec1afd3ac9f2f4839b5242479fc85fd7cjoin_yr%27
[2022-09-02 03:43:02,683] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.169466ms
[2022-09-02 03:43:02,683] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '2ede51e4-5846-4aa9-b1f2-2ff9a37c5671', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.169466ms', 'executionTime': '1.11794ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:43:02,786] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:02,789] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:02,789] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:03,380] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:03,435] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:43:03,435] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:43:03,533] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:43:03,587] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:03,590] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:03,590] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:04,207] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:04,267] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:43:04,267] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:43:04,372] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:43:04,430] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:43:04,430] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:43:04,430] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:43:04,457] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:43:04,483] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:43:04,491] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.270669ms
[2022-09-02 03:43:04,516] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:43:04,542] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:43:04,583] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.883249ms
[2022-09-02 03:43:04,655] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:43:04,656] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.9021964191466, 'mem_free': 12968845312, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:43:04,656] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:43:04,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:04,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:05,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:05,296] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:05,296] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:05,899] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:05,904] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:05,904] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:06,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:06,699] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:06,699] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:07,760] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:14,851] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:43:15,004] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:43:15,417] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:43:15,445] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:43:15,445] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:43:15,503] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,556] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,609] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:43:15,610] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:43:15,689] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:43:15,690] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:43:15,715] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,844] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:43:15,844] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,870] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:43:15,897] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:43:15,897] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,925] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:43:15,951] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:43:15,952] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:15,979] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:43:16,005] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:43:16,005] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:16,034] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:43:16,034] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #7 test_multi_create_drop_index ==============
[2022-09-02 03:43:16,034] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:43:16,035] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_7
ok
----------------------------------------------------------------------
Ran 1 test in 106.949s
OK
test_multi_create_drop_index (gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index,groups=composite,dataset=default,doc-per-day=1,cbq_version=sherlock,skip_build_tuq=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'composite', 'dataset': 'default', 'doc-per-day': '1', 'cbq_version': 'sherlock', 'skip_build_tuq': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 8, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8'}
[2022-09-02 03:43:16,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:16,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:16,771] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:16,804] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:43:16,884] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:43:16,884] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #8 test_multi_create_drop_index==============
[2022-09-02 03:43:16,885] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:43:17,158] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:43:17,188] - [task:164] INFO - {'uptime': '1006', 'memoryTotal': 15466930176, 'memoryFree': 13097017344, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:17,216] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:43:17,216] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:43:17,217] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:43:17,248] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:43:17,294] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:43:17,294] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:43:17,322] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:43:17,323] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:43:17,324] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:43:17,324] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:43:17,372] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:17,376] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:17,376] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:17,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:17,989] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:18,105] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:18,106] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:18,136] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:18,136] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:18,256] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:43:18,283] - [task:164] INFO - {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13097246720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:18,308] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:18,337] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:43:18,337] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:43:18,388] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:18,391] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:18,391] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:18,979] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:18,980] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:19,100] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:19,101] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:19,131] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:19,132] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:19,250] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:43:19,277] - [task:164] INFO - {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13097013248, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:19,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:19,330] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:43:19,331] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:43:19,381] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:19,386] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:19,386] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:19,976] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:19,977] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:20,094] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:20,095] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:20,130] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:20,131] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:20,247] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:43:20,274] - [task:164] INFO - {'uptime': '1003', 'memoryTotal': 15466930176, 'memoryFree': 13096919040, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:43:20,301] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:43:20,328] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:43:20,328] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:43:20,379] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:43:20,382] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:20,382] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:20,988] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:20,989] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:21,103] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:21,104] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:21,136] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:43:21,137] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:43:21,231] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:43:21,628] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:43:26,634] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:43:26,717] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:43:26,722] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:43:26,722] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:43:27,336] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:43:27,337] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:43:27,450] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:43:27,451] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:43:27,451] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:43:28,346] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:43:28,401] - [on_prem_rest_client:3047] INFO - 0.05 seconds to create bucket default
[2022-09-02 03:43:28,402] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:44:05,709] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:06,022] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:06,374] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:44:06,377] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:44:06,437] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:06,437] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:07,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:07,043] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:07,043] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:07,933] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:07,945] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:07,946] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:09,002] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:09,013] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:09,014] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:10,078] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:16,538] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:44:16,539] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.44980424605788, 'mem_free': 13095411712, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:44:16,539] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:44:16,539] - [basetestcase:467] INFO - Time to execute basesetup : 60.41486215591431
[2022-09-02 03:44:16,590] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,590] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:44:16,646] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,646] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:44:16,702] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,702] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:44:16,758] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:44:16,758] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:44:16,813] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:16,875] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:44:16,876] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:44:16,876] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:44:21,884] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:44:21,888] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:44:21,889] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:44:22,483] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:44:23,555] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:44:23,722] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:44:25,955] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:44:26,036] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:26,036] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:44:26,036] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:44:56,066] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:44:56,095] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:44:56,120] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:44:56,184] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 61.89703ms
[2022-09-02 03:44:56,184] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'aee697de-7379-4eeb-bda1-0bb835f46e7b', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '61.89703ms', 'executionTime': '61.833391ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:44:56,185] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:44:56,211] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:44:56,236] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:44:56,877] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 638.631252ms
[2022-09-02 03:44:56,877] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:44:56,951] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:44:56,994] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:44:57,006] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 8.918848ms
[2022-09-02 03:44:57,247] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:44:57,283] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:44:57,311] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:44:57,312] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:44:57,337] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:44:57,338] - [base_gsi:326] INFO - []
[2022-09-02 03:44:58,213] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee09c2890e7403479a81167a6a98ff8403job_title_join_yr` ON default(join_yr,job_title) WHERE job_title IS NOT NULL USING GSI WITH {'defer_build': True}
[2022-09-02 03:44:58,238] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%60+ON+default%28join_yr%2Cjob_title%29+WHERE++job_title+IS+NOT+NULL++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:44:58,280] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.570872ms
[2022-09-02 03:44:58,281] - [base_gsi:282] INFO - BUILD INDEX on default(employee09c2890e7403479a81167a6a98ff8403job_title_join_yr) USING GSI
[2022-09-02 03:44:59,311] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(employee09c2890e7403479a81167a6a98ff8403job_title_join_yr) USING GSI
[2022-09-02 03:44:59,337] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%29+USING+GSI
[2022-09-02 03:44:59,360] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 20.535152ms
[2022-09-02 03:45:00,392] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:00,418] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:00,427] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.561603ms
[2022-09-02 03:45:01,459] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:01,487] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:01,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.892542ms
[2022-09-02 03:45:02,527] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:02,555] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:02,564] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.447795ms
[2022-09-02 03:45:02,593] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee09c2890e7403479a81167a6a98ff8403job_title_join_yr ON default USING GSI
[2022-09-02 03:45:02,621] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee09c2890e7403479a81167a6a98ff8403job_title_join_yr+ON+default+USING+GSI
[2022-09-02 03:45:02,669] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 45.529735ms
[2022-09-02 03:45:02,704] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee09c2890e7403479a81167a6a98ff8403job_title_join_yr'
[2022-09-02 03:45:02,735] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee09c2890e7403479a81167a6a98ff8403job_title_join_yr%27
[2022-09-02 03:45:02,742] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.994653ms
[2022-09-02 03:45:02,743] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '2a562b68-8aca-4dde-bdf8-173679fed5da', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.994653ms', 'executionTime': '5.933741ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:45:02,852] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:02,856] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:02,856] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:03,485] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:03,541] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:45:03,541] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:45:03,641] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:45:03,696] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:03,700] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:03,700] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:04,324] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:04,382] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:45:04,383] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:45:04,484] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:45:04,540] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:45:04,541] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:45:04,541] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:45:04,567] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:45:04,593] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:45:04,600] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.4227ms
[2022-09-02 03:45:04,625] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:45:04,650] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:45:04,704] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 42.037635ms
[2022-09-02 03:45:04,775] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:45:04,775] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 11.91682580504829, 'mem_free': 12939608064, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:45:04,775] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:45:04,782] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:04,782] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:05,410] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:05,415] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:05,416] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:06,417] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:06,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:06,425] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:07,558] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:07,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:07,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:08,690] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:15,333] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:45:15,478] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:45:16,314] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:45:16,342] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:45:16,342] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:45:16,397] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,450] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,505] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:45:16,506] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:45:16,587] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:45:16,590] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:45:16,627] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,760] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:45:16,761] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,787] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:45:16,813] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:45:16,813] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,841] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:45:16,867] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:45:16,867] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,893] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:45:16,918] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:45:16,919] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:16,945] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:45:16,946] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #8 test_multi_create_drop_index ==============
[2022-09-02 03:45:16,946] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:45:16,947] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_8
ok
----------------------------------------------------------------------
Ran 1 test in 120.876s
OK
test_remove_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 9, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9'}
[2022-09-02 03:45:17,036] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:17,036] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:17,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:17,683] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:45:17,762] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:45:17,762] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #9 test_remove_bucket_and_query==============
[2022-09-02 03:45:17,763] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:45:18,062] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:45:18,095] - [task:164] INFO - {'uptime': '1127', 'memoryTotal': 15466930176, 'memoryFree': 13090742272, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:18,121] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:45:18,121] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:45:18,121] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:45:18,150] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:45:18,187] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:45:18,188] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:45:18,216] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:45:18,217] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:45:18,266] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:18,269] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:18,269] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:18,873] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:18,874] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:18,992] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:18,993] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:19,024] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:19,025] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:19,148] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:45:19,175] - [task:164] INFO - {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091569664, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:19,200] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:19,228] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:45:19,228] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:45:19,279] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:19,282] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:19,283] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:19,897] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:19,898] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:20,014] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:20,016] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:20,049] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:20,050] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:20,167] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:45:20,195] - [task:164] INFO - {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091950592, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:20,220] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:20,248] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:45:20,248] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:45:20,300] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:20,305] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:20,306] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:20,929] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:20,930] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:21,052] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:21,053] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:21,083] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:21,084] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:21,201] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:45:21,227] - [task:164] INFO - {'uptime': '1123', 'memoryTotal': 15466930176, 'memoryFree': 13091540992, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:45:21,253] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:45:21,281] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:45:21,282] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:45:21,337] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:45:21,340] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:21,340] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:21,941] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:21,942] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:22,062] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:22,063] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:22,093] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:45:22,093] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:45:22,185] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:45:22,578] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:45:27,582] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:45:27,666] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:45:27,671] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:45:27,674] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:45:28,302] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:45:28,304] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:45:28,423] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:45:28,424] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:45:28,424] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:45:29,307] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:45:29,363] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:45:29,363] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:46:05,612] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:05,939] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:06,208] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:46:06,212] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:46:06,278] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:06,278] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:06,981] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:06,986] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:06,987] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:07,954] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:07,969] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:07,969] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:09,061] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:09,070] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:09,070] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:10,166] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:16,708] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:46:16,710] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.17919900590499, 'mem_free': 13086642176, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:46:16,711] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:46:16,711] - [basetestcase:467] INFO - Time to execute basesetup : 59.67742037773132
[2022-09-02 03:46:16,762] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,763] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:46:16,817] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,817] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:46:16,870] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,871] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:46:16,928] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:46:16,928] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:46:16,983] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:17,047] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:46:17,047] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:46:17,047] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:46:22,058] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:46:22,062] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:46:22,063] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:46:22,686] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:46:23,858] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:46:24,049] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:46:27,176] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:46:27,256] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:27,256] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:46:27,257] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:46:57,286] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:46:57,315] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:46:57,352] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:46:57,424] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 69.478348ms
[2022-09-02 03:46:57,424] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '18739bd7-b20f-4dd1-a13c-d7cc46710ab1', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '69.478348ms', 'executionTime': '69.399192ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:46:57,424] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:46:57,452] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:46:57,479] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:46:58,163] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 681.9479ms
[2022-09-02 03:46:58,163] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:46:58,216] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:46:58,256] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:46:58,265] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.414397ms
[2022-09-02 03:46:58,480] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:58,525] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:46:58,555] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:46:58,555] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:46:58,571] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:46:58,642] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:46:59,440] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee4b6c1deee4d047558d38b2d6c2e111a3join_yr` ON default(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'defer_build': True}
[2022-09-02 03:46:59,465] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:46:59,495] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.224513ms
[2022-09-02 03:46:59,496] - [base_gsi:282] INFO - BUILD INDEX on default(`employee4b6c1deee4d047558d38b2d6c2e111a3join_yr`) USING GSI
[2022-09-02 03:47:00,528] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee4b6c1deee4d047558d38b2d6c2e111a3join_yr`) USING GSI
[2022-09-02 03:47:00,555] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%60%29+USING+GSI
[2022-09-02 03:47:00,581] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 22.520428ms
[2022-09-02 03:47:00,630] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:00,663] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:00,672] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.610843ms
[2022-09-02 03:47:01,705] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:01,732] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:01,740] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.933808ms
[2022-09-02 03:47:02,778] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:02,803] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:02,810] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.856132ms
[2022-09-02 03:47:03,841] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:47:03,868] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:47:03,872] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.890287ms
[2022-09-02 03:47:03,872] - [base_gsi:504] INFO - {'requestID': 'a92cb88c-8fbd-4f81-8fcc-ac9b19b69abd', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr', 'index_id': '9da2e12ce9deead3', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.890287ms', 'executionTime': '2.831206ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:47:03,873] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:47:03,873] - [tuq_generators:72] INFO - WHERE clause ===== is doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:47:03,874] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:47:03,874] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:47:03,875] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,875] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,876] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:47:03,876] - [tuq_generators:329] INFO - -->where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:47:03,935] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:47:03,979] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:47:04,005] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:47:04,190] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.079145ms
[2022-09-02 03:47:04,190] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:47:04,191] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:47:06,465] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:06,495] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee4b6c1deee4d047558d38b2d6c2e111a3join_yr'
[2022-09-02 03:47:06,541] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee4b6c1deee4d047558d38b2d6c2e111a3join_yr%27
[2022-09-02 03:47:06,547] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 4.778306ms
[2022-09-02 03:47:06,547] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '183cab59-02dc-43fe-85fb-5f5a2c485b99', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '4.778306ms', 'executionTime': '4.718788ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:47:06,657] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:06,661] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:06,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:07,429] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:07,489] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:47:07,490] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:47:07,602] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:47:07,659] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:07,662] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:07,662] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:08,356] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:08,415] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:47:08,415] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:47:08,529] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:47:08,593] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:47:08,593] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:47:08,593] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:47:08,620] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:47:08,646] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:47:08,653] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.814265ms
[2022-09-02 03:47:08,653] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '1206497e-c4d9-4a07-a312-9e24a27a146f', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.814265ms', 'executionTime': '5.754035ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:47:08,705] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:47:08,706] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 6.581473987757392, 'mem_free': 12942626816, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:47:08,706] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:47:08,710] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:08,710] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:09,359] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:09,364] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:09,364] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:10,241] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:10,250] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:10,250] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:11,436] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:11,449] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:11,450] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:12,624] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:19,845] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:47:19,986] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,040] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,093] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,146] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:47:20,147] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:47:20,224] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:47:20,225] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:47:20,250] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,381] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:47:20,381] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,408] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:47:20,434] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:47:20,434] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,460] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:47:20,486] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:47:20,486] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,512] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:47:20,539] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:47:20,539] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:20,567] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:47:20,567] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #9 test_remove_bucket_and_query ==============
[2022-09-02 03:47:20,568] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:47:20,568] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 1 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_9
ok
----------------------------------------------------------------------
Ran 1 test in 123.591s
OK
test_change_bucket_properties (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 10, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10'}
[2022-09-02 03:47:20,660] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:20,661] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:21,369] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:21,402] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:47:21,486] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:47:21,486] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #10 test_change_bucket_properties==============
[2022-09-02 03:47:21,487] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:47:21,687] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:47:21,713] - [task:164] INFO - {'uptime': '1252', 'memoryTotal': 15466930176, 'memoryFree': 13096235008, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:21,739] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:47:21,739] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:47:21,740] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:47:21,775] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:47:21,809] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:47:21,810] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:47:21,838] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:47:21,839] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:47:21,888] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:21,891] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:21,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:22,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:22,534] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:22,655] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:22,656] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:22,686] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:22,686] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:22,814] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:47:22,841] - [task:164] INFO - {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096726528, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:22,869] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:22,898] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:47:22,899] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:47:22,950] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:22,956] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:22,957] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:23,594] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:23,596] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:23,721] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:23,722] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:23,751] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:23,752] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:23,868] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:47:23,895] - [task:164] INFO - {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096243200, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:23,921] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:23,950] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:47:23,950] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:47:24,005] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:24,008] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:24,008] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:24,693] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:24,694] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:24,826] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:24,827] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:24,861] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:24,862] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:24,990] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:47:25,020] - [task:164] INFO - {'uptime': '1249', 'memoryTotal': 15466930176, 'memoryFree': 13096681472, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:47:25,046] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:47:25,075] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:47:25,076] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:47:25,130] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:47:25,132] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:25,133] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:25,823] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:25,824] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:25,956] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:25,957] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:25,987] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:47:25,988] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:47:26,080] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:47:26,455] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:47:31,461] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:47:31,556] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:47:31,560] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:47:31,561] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:47:32,247] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:47:32,249] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:47:32,382] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:47:32,383] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:47:32,383] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:47:33,241] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:47:33,297] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:47:33,298] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:48:05,840] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:06,155] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:06,443] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:48:06,447] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #10 test_change_bucket_properties ==============
[2022-09-02 03:48:06,512] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:06,512] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:07,162] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:07,167] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:07,167] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:08,161] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:08,179] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:08,180] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:09,339] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:09,351] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:09,352] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:10,533] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:17,462] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:48:17,463] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.68369701692233, 'mem_free': 13094109184, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:48:17,463] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:48:17,464] - [basetestcase:467] INFO - Time to execute basesetup : 56.80632734298706
[2022-09-02 03:48:17,518] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,518] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:48:17,574] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,574] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:48:17,628] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,628] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:48:17,685] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:48:17,685] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:48:17,739] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:17,807] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:48:17,807] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:48:17,807] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:48:22,820] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:48:22,823] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:48:22,824] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:48:23,527] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:48:24,731] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:48:24,905] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:48:27,945] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:48:28,028] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:28,028] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:48:28,029] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:48:58,055] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:48:58,086] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:48:58,111] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:48:58,176] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.220952ms
[2022-09-02 03:48:58,177] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '97ec36e0-2b17-40ca-9a6d-9250e7dab5ca', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.220952ms', 'executionTime': '63.156224ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:48:58,177] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:48:58,202] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:48:58,226] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:48:58,855] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 627.0798ms
[2022-09-02 03:48:58,855] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:48:58,925] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:48:58,960] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:48:58,970] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.560628ms
[2022-09-02 03:48:59,198] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:48:59,234] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:48:59,249] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:48:59,250] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:48:59,262] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:48:59,350] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:00,160] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employee6ec54c502be04606bd31701bd27b3a5ejoin_yr` ON default(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'defer_build': True}
[2022-09-02 03:49:00,186] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:49:00,220] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 31.381888ms
[2022-09-02 03:49:00,221] - [base_gsi:282] INFO - BUILD INDEX on default(`employee6ec54c502be04606bd31701bd27b3a5ejoin_yr`) USING GSI
[2022-09-02 03:49:01,252] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employee6ec54c502be04606bd31701bd27b3a5ejoin_yr`) USING GSI
[2022-09-02 03:49:01,278] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%60%29+USING+GSI
[2022-09-02 03:49:01,308] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 28.004146ms
[2022-09-02 03:49:01,349] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:01,385] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:01,394] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.097798ms
[2022-09-02 03:49:02,424] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:02,450] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:02,458] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.264989ms
[2022-09-02 03:49:03,489] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:03,515] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:03,524] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.661173ms
[2022-09-02 03:49:04,555] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:49:04,584] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:49:04,588] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 2.872032ms
[2022-09-02 03:49:04,588] - [base_gsi:504] INFO - {'requestID': '20ea87d2-831f-48cb-a402-0ec19f6f57c0', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr', 'index_id': '477ca682526371b0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '2.872032ms', 'executionTime': '2.797905ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:49:04,589] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:49:04,589] - [tuq_generators:72] INFO - WHERE clause ===== is doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:04,590] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:49:04,590] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:49:04,590] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,590] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,591] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:04,591] - [tuq_generators:329] INFO - -->where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:04,652] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:04,693] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:49:04,719] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:49:04,889] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 158.301809ms
[2022-09-02 03:49:04,890] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:49:04,890] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:49:06,136] - [on_prem_rest_client:3084] INFO - http://127.0.0.1:9000/pools/default/buckets/default with param:
[2022-09-02 03:49:06,194] - [on_prem_rest_client:3092] INFO - bucket default updated
[2022-09-02 03:49:06,224] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:49:06,251] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:49:06,254] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.875253ms
[2022-09-02 03:49:06,254] - [base_gsi:504] INFO - {'requestID': 'c474215e-ea15-4e65-ba28-891de89aed5b', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr', 'index_id': '477ca682526371b0', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '1.875253ms', 'executionTime': '1.813943ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:49:06,255] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:49:06,255] - [tuq_generators:72] INFO - WHERE clause ===== is doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:06,255] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:49:06,255] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:49:06,256] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,256] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,256] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:06,256] - [tuq_generators:329] INFO - -->where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:49:06,308] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:49:06,346] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:49:06,371] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:49:06,463] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 83.048508ms
[2022-09-02 03:49:06,463] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:49:06,464] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:49:07,741] - [tuq_helper:320] INFO - RUN QUERY DROP INDEX employee6ec54c502be04606bd31701bd27b3a5ejoin_yr ON default USING GSI
[2022-09-02 03:49:07,768] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+INDEX+employee6ec54c502be04606bd31701bd27b3a5ejoin_yr+ON+default+USING+GSI
[2022-09-02 03:49:07,814] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 43.725086ms
[2022-09-02 03:49:07,848] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employee6ec54c502be04606bd31701bd27b3a5ejoin_yr'
[2022-09-02 03:49:07,877] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employee6ec54c502be04606bd31701bd27b3a5ejoin_yr%27
[2022-09-02 03:49:07,884] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.484406ms
[2022-09-02 03:49:07,884] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '9df9d843-3f98-4a3b-981c-7aa21335d831', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.484406ms', 'executionTime': '5.423583ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:49:07,988] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:07,991] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:07,991] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:08,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:08,740] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:49:08,740] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:49:08,857] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:49:08,917] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:08,920] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:08,920] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:09,658] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:09,719] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:49:09,719] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:49:09,836] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:49:09,893] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:49:09,893] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:49:09,893] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:49:09,919] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:49:09,945] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:49:09,952] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.557482ms
[2022-09-02 03:49:09,977] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:49:10,003] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:49:10,051] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 46.389252ms
[2022-09-02 03:49:10,119] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:49:10,119] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.83201563203367, 'mem_free': 12944936960, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:49:10,120] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:49:10,123] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:10,124] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:10,821] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:10,826] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:10,826] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:11,479] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:11,484] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:11,484] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:12,557] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:12,565] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:12,566] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:13,776] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:21,566] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #10 test_change_bucket_properties ==============
[2022-09-02 03:49:21,712] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:49:22,359] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:49:22,387] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:49:22,387] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:49:22,441] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,493] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,546] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:49:22,547] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:49:22,626] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:49:22,627] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:49:22,653] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,794] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:49:22,794] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,822] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:49:22,860] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:49:22,860] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,887] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:49:22,916] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:49:22,916] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,943] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:49:22,969] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:49:22,969] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:22,998] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:49:22,998] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #10 test_change_bucket_properties ==============
[2022-09-02 03:49:22,998] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:49:22,999] - [basetestcase:778] INFO - closing all memcached connections
Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 2 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_10
ok
----------------------------------------------------------------------
Ran 1 test in 122.396s
OK
test_delete_create_bucket_and_query (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Logs will be stored at /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11
./testrunner -i b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini -p makefile=True,gsi_type=memory_optimized -t gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query,groups=simple:and:no_orderby_groupby:range,dataset=default,doc-per-day=1,use_gsi_for_primary=True,use_gsi_for_secondary=True,GROUP=gsi
Test Input params:
{'groups': 'simple:and:no_orderby_groupby:range', 'dataset': 'default', 'doc-per-day': '1', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'GROUP': 'gsi', 'ini': 'b/resources/dev-4-nodes-xdcr_n1ql_gsi.ini', 'cluster_name': 'dev-4-nodes-xdcr_n1ql_gsi', 'spec': 'simple_gsi_n1ql', 'conf_file': 'conf/simple_gsi_n1ql.conf', 'makefile': 'True', 'gsi_type': 'memory_optimized', 'num_nodes': 4, 'case_number': 11, 'total_testcases': 11, 'last_case_fail': 'False', 'teardown_run': 'True', 'logs_folder': '/opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11'}
[2022-09-02 03:49:23,089] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:23,089] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:23,755] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:23,787] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:49:23,867] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:49:23,867] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #11 test_delete_create_bucket_and_query==============
[2022-09-02 03:49:23,868] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:49:24,120] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:49:24,147] - [task:164] INFO - {'uptime': '1373', 'memoryTotal': 15466930176, 'memoryFree': 13081886720, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:24,173] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:49:24,174] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:49:24,174] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:49:24,207] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:49:24,243] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:49:24,244] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:49:24,271] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:49:24,272] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:49:24,272] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:49:24,273] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:49:24,324] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:24,327] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:24,328] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:25,001] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:25,002] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:25,132] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:25,133] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:25,163] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:25,163] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:25,288] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:49:25,320] - [task:164] INFO - {'uptime': '1369', 'memoryTotal': 15466930176, 'memoryFree': 13081690112, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:25,347] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:25,376] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:49:25,377] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:49:25,432] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:25,435] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:25,436] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:26,164] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:26,165] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:26,307] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:26,308] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:26,342] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:26,342] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:26,470] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:49:26,499] - [task:164] INFO - {'uptime': '1370', 'memoryTotal': 15466930176, 'memoryFree': 13082599424, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:26,528] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:26,557] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:49:26,558] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:49:26,611] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:26,617] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:26,617] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:27,340] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:27,341] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:27,484] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:27,485] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:27,519] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:27,519] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:27,640] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:49:27,667] - [task:164] INFO - {'uptime': '1369', 'memoryTotal': 15466930176, 'memoryFree': 13082394624, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:49:27,695] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:49:27,724] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:49:27,725] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:49:27,782] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:49:27,785] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:27,785] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:28,472] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:28,473] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:28,600] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:28,601] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:28,632] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:49:28,632] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:49:28,723] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:49:29,105] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:49:34,110] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:49:34,201] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:49:34,207] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:49:34,208] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:49:34,907] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:49:34,908] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:49:35,043] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:49:35,044] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:49:35,045] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:49:35,902] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:49:35,960] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:49:35,960] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:50:05,766] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:06,079] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:06,341] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:50:06,344] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:50:06,396] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:06,396] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:07,106] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:07,112] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:07,112] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:08,173] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:08,182] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:08,182] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:09,413] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:09,422] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:09,422] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:10,614] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:17,888] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:50:17,889] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 12.73216744020807, 'mem_free': 13078851584, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:50:17,889] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:50:17,889] - [basetestcase:467] INFO - Time to execute basesetup : 54.80256128311157
[2022-09-02 03:50:17,944] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:17,944] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:50:18,001] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,001] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:50:18,054] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,054] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:50:18,109] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:50:18,109] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:50:18,161] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:18,223] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:50:18,223] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:50:18,224] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:50:23,236] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:50:23,239] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:50:23,240] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:50:23,970] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:50:25,186] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:50:25,374] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:50:28,330] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:50:28,412] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:28,413] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:50:28,413] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:50:58,439] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:50:58,468] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:50:58,493] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:50:58,559] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 63.929771ms
[2022-09-02 03:50:58,559] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '52b5abb4-6cad-4518-8f52-7ee3e81a3160', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '63.929771ms', 'executionTime': '63.859059ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:50:58,559] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:50:58,585] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:50:58,610] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:50:59,308] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 696.545099ms
[2022-09-02 03:50:59,309] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:50:59,365] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:50:59,402] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:50:59,411] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.129552ms
[2022-09-02 03:50:59,614] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:50:59,651] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:50:59,671] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:50:59,674] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:50:59,693] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:50:59,772] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:51:00,573] - [tuq_helper:320] INFO - RUN QUERY CREATE INDEX `employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr` ON default(join_yr) WHERE join_yr > 2010 and join_yr < 2014 USING GSI WITH {'defer_build': True}
[2022-09-02 03:51:00,599] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+INDEX+%60employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%60+ON+default%28join_yr%29+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014++USING+GSI++WITH+%7B%27defer_build%27%3A+True%7D
[2022-09-02 03:51:00,641] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 39.226025ms
[2022-09-02 03:51:00,641] - [base_gsi:282] INFO - BUILD INDEX on default(`employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr`) USING GSI
[2022-09-02 03:51:01,673] - [tuq_helper:320] INFO - RUN QUERY BUILD INDEX on default(`employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr`) USING GSI
[2022-09-02 03:51:01,701] - [on_prem_rest_client:4201] INFO - query params : statement=BUILD+INDEX+on+default%28%60employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%60%29+USING+GSI
[2022-09-02 03:51:01,724] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 21.175858ms
[2022-09-02 03:51:01,780] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:01,811] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:01,820] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 7.523242ms
[2022-09-02 03:51:02,851] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:02,876] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:02,883] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.610763ms
[2022-09-02 03:51:03,914] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:51:03,942] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:51:03,951] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 6.667847ms
[2022-09-02 03:51:04,982] - [tuq_helper:320] INFO - RUN QUERY EXPLAIN SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:51:05,007] - [on_prem_rest_client:4201] INFO - query params : statement=EXPLAIN+SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+
[2022-09-02 03:51:05,012] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 3.591706ms
[2022-09-02 03:51:05,013] - [base_gsi:504] INFO - {'requestID': '021939d3-bbe2-4633-a6ec-e6d323cd140e', 'signature': 'json', 'results': [{'plan': {'#operator': 'Sequence', '~children': [{'#operator': 'Sequence', '~children': [{'#operator': 'IndexScan3', 'index': 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr', 'index_id': '3064d7a6164a52a4', 'index_projection': {'primary_key': True}, 'keyspace': 'default', 'namespace': 'default', 'spans': [{'exact': True, 'range': [{'high': '2014', 'inclusion': 0, 'index_key': '`join_yr`', 'low': '2010'}]}], 'using': 'gsi'}, {'#operator': 'Fetch', 'keyspace': 'default', 'namespace': 'default'}, {'#operator': 'Parallel', '~child': {'#operator': 'Sequence', '~children': [{'#operator': 'Filter', 'condition': '((2010 < (`default`.`join_yr`)) and ((`default`.`join_yr`) < 2014))'}, {'#operator': 'InitialProject', 'result_terms': [{'expr': 'self', 'star': True}]}]}}]}, {'#operator': 'Order', 'can_spill': True, 'clip_values': True, 'sort_terms': [{'expr': '(`default`.`_id`)'}]}]}, 'text': 'SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id'}], 'status': 'success', 'metrics': {'elapsedTime': '3.591706ms', 'executionTime': '3.524317ms', 'resultCount': 1, 'resultSize': 926, 'serviceLoad': 6}}
[2022-09-02 03:51:05,013] - [tuq_generators:70] INFO - FROM clause ===== is default
[2022-09-02 03:51:05,013] - [tuq_generators:72] INFO - WHERE clause ===== is doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:51:05,014] - [tuq_generators:74] INFO - UNNEST clause ===== is None
[2022-09-02 03:51:05,014] - [tuq_generators:76] INFO - SELECT clause ===== is {"*" : doc,}
[2022-09-02 03:51:05,014] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,014] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,015] - [tuq_generators:326] INFO - -->select_clause:{"*" : doc,"_id" : doc["_id"]}; where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:51:05,015] - [tuq_generators:329] INFO - -->where_clause= doc["join_yr"] > 2010 and doc["join_yr"] < 2014
[2022-09-02 03:51:05,074] - [tuq_generators:419] INFO - ORDER clause ========= is doc["_id"],
[2022-09-02 03:51:05,112] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM default WHERE join_yr > 2010 and join_yr < 2014 ORDER BY _id
[2022-09-02 03:51:05,138] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+default+WHERE++join_yr+%3E+2010+and+join_yr+%3C+2014+ORDER+BY+_id+&scan_consistency=request_plus
[2022-09-02 03:51:05,323] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 172.385662ms
[2022-09-02 03:51:05,323] - [tuq_helper:409] INFO - Analyzing Actual Result
[2022-09-02 03:51:05,324] - [tuq_helper:411] INFO - Analyzing Expected Result
[2022-09-02 03:51:07,325] - [basetestcase:847] INFO - sleep for 2 secs. ...
[2022-09-02 03:51:10,062] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:51:10,124] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:51:10,124] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:52:05,569] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:52:05,888] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:52:06,288] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:52:06,420] - [basetestcase:847] INFO - sleep for 2 secs. ...
[2022-09-02 03:52:08,457] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:52:08,458] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,487] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:52:08,517] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:52:08,517] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,545] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:52:08,574] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:52:08,575] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,603] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:52:08,631] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:52:08,631] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:08,658] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:52:08,710] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:08,736] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:52:08,762] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:52:08,804] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.436456ms
[2022-09-02 03:52:08,804] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'b1963f3d-d268-4e4e-aeeb-a84a569125d9', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '40.436456ms', 'executionTime': '40.377327ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:08,832] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = 'employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr'
[2022-09-02 03:52:08,858] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27employeef07bacc4b8b84764a6f8afe38931a5cejoin_yr%27
[2022-09-02 03:52:08,861] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.082086ms
[2022-09-02 03:52:08,861] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '8d5315ac-46f9-431e-a51e-c3d7d4549d30', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.082086ms', 'executionTime': '1.02148ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:08,916] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:09,046] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:09,047] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:09,048] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:09,766] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:09,827] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:52:09,828] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:52:09,942] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:52:09,998] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:10,001] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:10,002] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:10,727] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:10,785] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:52:10,786] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:52:10,906] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:52:10,967] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:52:10,967] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:52:10,967] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:52:10,993] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:52:11,020] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:52:11,027] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.1186ms
[2022-09-02 03:52:11,027] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '2bd56144-eeb5-4825-9bf5-c42b42a83f45', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '5.1186ms', 'executionTime': '5.04852ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:11,027] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:52:11,055] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:52:11,083] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:52:11,086] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 1.381899ms
[2022-09-02 03:52:11,087] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': '6299aac7-ea6f-4fe4-9a30-c6d8a98f07f5', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '1.381899ms', 'executionTime': '1.312904ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:52:11,146] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:52:11,146] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 8.060210697253556, 'mem_free': 13067821056, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:52:11,147] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:52:11,151] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:11,151] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:11,876] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:11,881] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:11,881] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:12,797] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:12,809] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:12,810] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:14,055] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:14,067] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:14,068] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:15,285] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:22,994] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:52:23,146] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:52:23,464] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:52:23,494] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:52:23,494] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:52:23,552] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,607] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,663] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:52:23,664] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:52:23,750] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:52:23,751] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:52:23,778] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:23,917] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:52:23,917] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:23,947] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:52:23,975] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:52:23,975] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,004] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:52:24,030] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:52:24,030] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,059] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:52:24,088] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:52:24,088] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:24,118] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:52:24,118] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #11 test_delete_create_bucket_and_query ==============
[2022-09-02 03:52:24,119] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:52:24,119] - [basetestcase:778] INFO - closing all memcached connections
ok
----------------------------------------------------------------------
Ran 1 test in 181.091s
OK
suite_tearDown (gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests) ... Cluster instance shutdown with force
summary so far suite gsi.indexscans_gsi.SecondaryIndexingScanTests , pass 6 , fail 0
summary so far suite gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests , pass 2 , fail 0
summary so far suite gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests , pass 3 , fail 0
testrunner logs, diags and results are available under /opt/build/testrunner/logs/testrunner-22-Sep-02_03-26-26/test_11
*** Tests executed count: 11
Run after suite setup for gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query
[2022-09-02 03:52:24,185] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:24,185] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:24,961] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:24,996] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:52:25,079] - [on_prem_rest_client:2668] INFO - Node version in cluster 7.2.0-1948-rel-EE-enterprise
[2022-09-02 03:52:25,080] - [basetestcase:157] INFO - ============== basetestcase setup was started for test #11 suite_tearDown==============
[2022-09-02 03:52:25,080] - [basetestcase:224] INFO - initializing cluster
[2022-09-02 03:52:25,213] - [task:159] INFO - server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self
[2022-09-02 03:52:25,242] - [task:164] INFO - {'uptime': '1554', 'memoryTotal': 15466930176, 'memoryFree': 13065805824, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9000', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7650, 'memcached': 12000, 'id': 'n_0@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9000', 'services': ['index', 'kv', 'n1ql'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:25,270] - [task:203] INFO - quota for index service will be 256 MB
[2022-09-02 03:52:25,270] - [task:205] INFO - set index quota to node 127.0.0.1
[2022-09-02 03:52:25,271] - [on_prem_rest_client:1271] INFO - pools/default params : indexMemoryQuota=256
[2022-09-02 03:52:25,303] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7650
[2022-09-02 03:52:25,343] - [on_prem_rest_client:1218] INFO - --> init_node_services(Administrator,asdasd,127.0.0.1,9000,['kv', 'index', 'n1ql'])
[2022-09-02 03:52:25,344] - [on_prem_rest_client:1234] INFO - /node/controller/setupServices params on 127.0.0.1: 9000:hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql
[2022-09-02 03:52:25,372] - [on_prem_rest_client:1080] ERROR - POST http://127.0.0.1:9000//node/controller/setupServices body: hostname=127.0.0.1%3A9000&user=Administrator&password=asdasd&services=kv%2Cindex%2Cn1ql headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 400 reason: unknown b'{"errors":{"services":"cannot change node services after cluster is provisioned"}}' auth: Administrator:asdasd
[2022-09-02 03:52:25,373] - [on_prem_rest_client:1240] INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2022-09-02 03:52:25,373] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9000
[2022-09-02 03:52:25,374] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9000:port=9000&username=Administrator&password=asdasd
[2022-09-02 03:52:25,422] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:25,425] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:25,426] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:26,135] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:26,136] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:26,271] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:26,273] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:26,305] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:26,306] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:26,425] - [task:159] INFO - server: ip:127.0.0.1 port:9001 ssh_username:Administrator, nodes/self
[2022-09-02 03:52:26,453] - [task:164] INFO - {'uptime': '1555', 'memoryTotal': 15466930176, 'memoryFree': 13066162176, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9001', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12002, 'id': 'n_1@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9001', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:26,479] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:26,507] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9001
[2022-09-02 03:52:26,508] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9001:port=9001&username=Administrator&password=asdasd
[2022-09-02 03:52:26,560] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:26,563] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:26,564] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:27,290] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:27,291] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9001/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:27,421] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:27,422] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:27,453] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9001: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:27,454] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:27,572] - [task:159] INFO - server: ip:127.0.0.1 port:9002 ssh_username:Administrator, nodes/self
[2022-09-02 03:52:27,599] - [task:164] INFO - {'uptime': '1550', 'memoryTotal': 15466930176, 'memoryFree': 13064998912, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9002', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12004, 'id': 'n_2@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9002', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:27,625] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:27,653] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9002
[2022-09-02 03:52:27,654] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9002:port=9002&username=Administrator&password=asdasd
[2022-09-02 03:52:27,706] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:27,709] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:27,709] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:28,442] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:28,443] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9002/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:28,588] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:28,589] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:28,620] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9002: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:28,621] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:28,746] - [task:159] INFO - server: ip:127.0.0.1 port:9003 ssh_username:Administrator, nodes/self
[2022-09-02 03:52:28,776] - [task:164] INFO - {'uptime': '1550', 'memoryTotal': 15466930176, 'memoryFree': 13058134016, 'mcdMemoryReserved': 11800, 'mcdMemoryAllocated': 11800, 'status': 'healthy', 'hostname': '127.0.0.1:9003', 'clusterCompatibility': 458754, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.2.0-1948-rel-EE-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 7906, 'memcached': 12006, 'id': 'n_3@cb.local', 'ip': '127.0.0.1', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '9003', 'services': ['kv'], 'storageTotalRam': 14750, 'curr_items': 0}
[2022-09-02 03:52:28,803] - [on_prem_rest_client:1258] INFO - pools/default params : memoryQuota=7906
[2022-09-02 03:52:28,834] - [on_prem_rest_client:1152] INFO - --> in init_cluster...Administrator,asdasd,9003
[2022-09-02 03:52:28,834] - [on_prem_rest_client:1157] INFO - settings/web params on 127.0.0.1:9003:port=9003&username=Administrator&password=asdasd
[2022-09-02 03:52:28,887] - [on_prem_rest_client:1159] INFO - --> status:True
[2022-09-02 03:52:28,890] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:28,891] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:29,587] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:29,588] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9003/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:29,728] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:29,729] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:29,760] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9003: True content: [7,2] command: cluster_compat_mode:get_compat_version().
[2022-09-02 03:52:29,761] - [on_prem_rest_client:1295] INFO - settings/indexes params : storageMode=memory_optimized
[2022-09-02 03:52:29,864] - [basetestcase:2396] INFO - **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
[2022-09-02 03:52:30,258] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:52:35,260] - [basetestcase:2401] INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2022-09-02 03:52:35,346] - [basetestcase:262] INFO - done initializing cluster
[2022-09-02 03:52:35,350] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:52:35,350] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:52:36,052] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:52:36,053] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: curl --silent --show-error http://Administrator:asdasd@localhost:9000/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2022-09-02 03:52:36,194] - [remote_util:3397] INFO - command executed with Administrator but got an error b'curl: /opt/build/install/lib/libcurl.so.4: no version information available (required by curl)\n' ...
[2022-09-02 03:52:36,195] - [remote_util:5249] INFO - b'ok'
[2022-09-02 03:52:36,196] - [basetestcase:2992] INFO - Enabled diag/eval for non-local hosts from 127.0.0.1
[2022-09-02 03:52:37,045] - [on_prem_rest_client:3022] INFO - http://127.0.0.1:9000/pools/default/buckets with param: name=default&ramQuotaMB=7650&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore
[2022-09-02 03:52:37,104] - [on_prem_rest_client:3047] INFO - 0.06 seconds to create bucket default
[2022-09-02 03:52:37,105] - [bucket_helper:340] INFO - waiting for memcached bucket : default in 127.0.0.1 to accept set ops
[2022-09-02 03:53:05,846] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:06,171] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:06,454] - [task:382] INFO - bucket 'default' was created with per node RAM quota: 7650
[2022-09-02 03:53:06,457] - [basetestcase:435] INFO - ============== basetestcase setup was finished for test #11 suite_tearDown ==============
[2022-09-02 03:53:06,515] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:06,515] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:07,266] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:07,271] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:07,271] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:08,449] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:08,460] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:08,461] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:09,728] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:09,736] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:09,736] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:11,012] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:18,460] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:53:18,460] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 13.15422214916433, 'mem_free': 13059473408, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:53:18,460] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:53:18,461] - [basetestcase:467] INFO - Time to execute basesetup : 54.27910399436951
[2022-09-02 03:53:18,513] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,513] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:53:18,571] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,571] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:53:18,625] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,625] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:53:18,679] - [newtuq:23] INFO - Initial status of 127.0.0.1 cluster is healthy
[2022-09-02 03:53:18,680] - [newtuq:28] INFO - current status of 127.0.0.1 is healthy
[2022-09-02 03:53:18,735] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:53:18,792] - [on_prem_rest_client:2093] INFO - {'indexer.settings.storage_mode': 'memory_optimized'} set
[2022-09-02 03:53:18,793] - [newtuq:39] INFO - Allowing the indexer to complete restart after setting the internal settings
[2022-09-02 03:53:18,793] - [basetestcase:847] INFO - sleep for 5 secs. ...
[2022-09-02 03:53:23,805] - [on_prem_rest_client:2093] INFO - {'indexer.api.enableTestServer': True} set
[2022-09-02 03:53:23,809] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:53:23,809] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:53:24,515] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:53:25,610] - [basetestcase:2772] INFO - create 2016.0 to default documents...
[2022-09-02 03:53:25,780] - [data_helper:312] INFO - creating direct client 127.0.0.1:12000 default
[2022-09-02 03:53:28,874] - [basetestcase:2785] INFO - LOAD IS FINISHED
[2022-09-02 03:53:28,951] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:53:28,951] - [newtuq:97] INFO - ip:127.0.0.1 port:9000 ssh_username:Administrator
[2022-09-02 03:53:28,952] - [basetestcase:847] INFO - sleep for 30 secs. ...
[2022-09-02 03:53:58,986] - [tuq_helper:755] INFO - Check if index existed in default on server 127.0.0.1
[2022-09-02 03:53:59,016] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:53:59,042] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:53:59,112] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 68.255591ms
[2022-09-02 03:53:59,113] - [tuq_helper:919] ERROR - Fail to get index list. List output: {'requestID': 'db60f5c0-5c8e-4943-8f77-5fc6e2186805', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '68.255591ms', 'executionTime': '68.193083ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}}
[2022-09-02 03:53:59,113] - [tuq_helper:758] INFO - Create primary index
[2022-09-02 03:53:59,139] - [tuq_helper:320] INFO - RUN QUERY CREATE PRIMARY INDEX ON default
[2022-09-02 03:53:59,165] - [on_prem_rest_client:4201] INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+default+
[2022-09-02 03:53:59,837] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 670.40011ms
[2022-09-02 03:53:59,838] - [tuq_helper:760] INFO - Check if index is online
[2022-09-02 03:53:59,913] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:53:59,946] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:53:59,953] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.874546ms
[2022-09-02 03:54:00,172] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,215] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"queryport.client.waitForScheduledIndex": false} client_cert=None verify=False
[2022-09-02 03:54:00,250] - [on_prem_rest_client:2080] INFO - {'queryport.client.waitForScheduledIndex': False} set
[2022-09-02 03:54:00,251] - [on_prem_rest_client:1116] INFO - Making a rest request api=http://127.0.0.1:9102/settings verb=POST params={"indexer.allowScheduleCreateRebal": true} client_cert=None verify=False
[2022-09-02 03:54:00,269] - [on_prem_rest_client:2080] INFO - {'indexer.allowScheduleCreateRebal': True} set
[2022-09-02 03:54:00,331] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,464] - [basetestcase:2663] INFO - list of index nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:00,467] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:00,467] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:01,307] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:01,365] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:54:01,365] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/indexer.log* | wc -l
[2022-09-02 03:54:01,494] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:54:01,551] - [basetestcase:2663] INFO - list of kv nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:01,554] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:01,554] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:02,376] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:02,436] - [on_prem_rest_client:1880] INFO - /diag/eval status on 127.0.0.1:9000: True content: "/opt/build/ns_server/logs/n_0" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2022-09-02 03:54:02,436] - [remote_util:3350] INFO - running command.raw on 127.0.0.1: zgrep "panic" "/opt/build/ns_server/logs/n_0"/projector.log* | wc -l
[2022-09-02 03:54:02,578] - [remote_util:3399] INFO - command executed successfully with Administrator
[2022-09-02 03:54:02,635] - [basetestcase:2663] INFO - list of n1ql nodes in cluster: [ip:127.0.0.1 port:9000 ssh_username:Administrator]
[2022-09-02 03:54:02,635] - [tuq_helper:731] INFO - CHECK FOR PRIMARY INDEXES
[2022-09-02 03:54:02,635] - [tuq_helper:738] INFO - DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:54:02,660] - [tuq_helper:320] INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary'
[2022-09-02 03:54:02,688] - [on_prem_rest_client:4201] INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27
[2022-09-02 03:54:02,695] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 5.782303ms
[2022-09-02 03:54:02,723] - [tuq_helper:320] INFO - RUN QUERY DROP PRIMARY INDEX ON default USING GSI
[2022-09-02 03:54:02,752] - [on_prem_rest_client:4201] INFO - query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI
[2022-09-02 03:54:02,794] - [tuq_helper:349] INFO - TOTAL ELAPSED TIME: 40.686858ms
[2022-09-02 03:54:02,864] - [basetestcase:601] INFO - ------- Cluster statistics -------
[2022-09-02 03:54:02,864] - [basetestcase:603] INFO - 127.0.0.1:9000 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 5.635379814318155, 'mem_free': 12891725824, 'mem_total': 15466930176, 'swap_mem_used': 0, 'swap_mem_total': 0}
[2022-09-02 03:54:02,864] - [basetestcase:604] INFO - --- End of cluster statistics ---
[2022-09-02 03:54:02,869] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:02,869] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:03,650] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:03,655] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:03,655] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:04,459] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:04,464] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:04,464] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:05,684] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:05,696] - [remote_util:318] INFO - SSH Connecting to 127.0.0.1 with username:Administrator, attempt#1 of 5
[2022-09-02 03:54:05,696] - [remote_util:354] INFO - SSH Connected to 127.0.0.1 as Administrator
[2022-09-02 03:54:07,038] - [remote_util:3674] INFO - extract_remote_info-->distribution_type: linux, distribution_version: default
[2022-09-02 03:54:15,919] - [basetestcase:719] INFO - ============== basetestcase cleanup was started for test #11 suite_tearDown ==============
[2022-09-02 03:54:16,285] - [bucket_helper:130] INFO - deleting existing buckets ['default'] on 127.0.0.1
[2022-09-02 03:54:17,390] - [bucket_helper:221] INFO - waiting for bucket deletion to complete....
[2022-09-02 03:54:17,430] - [on_prem_rest_client:141] INFO - node 127.0.0.1 existing buckets : []
[2022-09-02 03:54:17,430] - [bucket_helper:153] INFO - deleted bucket : default from 127.0.0.1
[2022-09-02 03:54:17,489] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,544] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,603] - [bucket_helper:155] INFO - Could not find any buckets for node 127.0.0.1, nothing to delete
[2022-09-02 03:54:17,604] - [basetestcase:738] INFO - Removing user 'clientuser'...
[2022-09-02 03:54:17,684] - [on_prem_rest_client:1080] ERROR - DELETE http://127.0.0.1:9000/settings/rbac/users/local/clientuser body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:asdasd
[2022-09-02 03:54:17,685] - [basetestcase:742] INFO - b'"User was not found."'
[2022-09-02 03:54:17,714] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,846] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9000
[2022-09-02 03:54:17,846] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,874] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9000 is running
[2022-09-02 03:54:17,901] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9001
[2022-09-02 03:54:17,901] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,929] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9001 is running
[2022-09-02 03:54:17,955] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9002
[2022-09-02 03:54:17,955] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:17,982] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9002 is running
[2022-09-02 03:54:18,008] - [cluster_helper:90] INFO - waiting for ns_server @ 127.0.0.1:9003
[2022-09-02 03:54:18,008] - [on_prem_rest_client:45] INFO - -->is_ns_server_running?
[2022-09-02 03:54:18,035] - [cluster_helper:94] INFO - ns_server @ 127.0.0.1:9003 is running
[2022-09-02 03:54:18,036] - [basetestcase:761] INFO - ============== basetestcase cleanup was finished for test #11 suite_tearDown ==============
[2022-09-02 03:54:18,036] - [basetestcase:773] INFO - closing all ssh connections
[2022-09-02 03:54:18,036] - [basetestcase:778] INFO - closing all memcached connections
ok
----------------------------------------------------------------------
Ran 1 test in 113.929s
OK
Cluster instance shutdown with force
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('Thread', , 'was not properly terminated, will be terminated now.')
Shutting down the thread...
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexscans_gsi.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.indexcreatedrop_gsi.SecondaryIndexingCreateDropTests.test_multi_create_drop_index', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_remove_bucket_and_query', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_change_bucket_properties', ' pass')
('gsi.cluster_ops_gsi.SecondaryIndexingClusterOpsTests.test_delete_create_bucket_and_query', ' pass')
*** TestRunner ***
scripts/start_cluster_and_run_tests.sh: line 91: 6055 Terminated COUCHBASE_NUM_VBUCKETS=64 python3 ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log (wd: /opt/build/ns_server)
Testing Failed: Required test failed
Binary file /opt/build/test.log matches
Binary file /opt/build/test.log matches
Version: versions-02.09.2022-01.46.cfg
Build Log: make-02.09.2022-01.46.log
Server Log: logs-02.09.2022-01.46.tar.gz
Finished