[CDCSDK] flaky test CDCSDKConsistentSnapshotTest.InsertBeforeAfterSnapshot

[jasonyb]

Jira Link: DB-11943

Description

Almalinux 8 ./yb_build.sh fastdebug --gcc11 --cxx-test integration-tests_cdcsdk_consistent_snapshot-test --gtest_filter CDCSDKConsistentSnapshotTest.InsertBeforeAfterSnapshot --tp 1 -n 50 fails on master 7217b58 and pg15 2722fc5 4 and 3 times, respectively.

Failures look like

Expected equality of these values:
  key
    Which is: 0
  record.row_message().new_tuple(0).datum_int32()
    Which is: 2
[ts-2] I0625 13:07:20.199236 4020339 tablet_limits.cc:62] vlog1: Tablet replica limit is 9223372036854775807
[ts-1] I0625 13:07:20.200168 4020256 tablet_service.cc:1186] vlog1: UpdateTransaction: tablet_id: "36a028a43b4148fabbd8ffb235bfca94" state { transaction_id: "\211\307\026w\013\260N\026\212Z^u\213\374:\024" status: PENDING host_node_uuid: "2033dc7c33174c239eff8992df1e0cb3" } propagated_hybrid_time: 7042441380659281920, context: Local call. call-id : 0
[ts-3] I0625 13:07:20.311106 4020383 tablet_limits.cc:62] vlog1: Tablet replica limit is 9223372036854775807
Test failure stack trace:
../../src/yb/integration-tests/cdcsdk_ysql_test_base.cc:1184:                                           @     0x7f21bde1db70  yb::cdc::CDCSDKYsqlTest::AssertKeyValue(yb::cdc::CDCSDKProtoRecordPB const&, int const&, int const&, bool const&, int const&)
../../src/yb/integration-tests/cdcsdk_ysql_test_base.cc:1295:                                           @     0x7f21bde3d954  yb::cdc::CDCSDKYsqlTest::CheckRecord(yb::cdc::CDCSDKProtoRecordPB const&, yb::cdc::CDCSDKYsqlTest::ExpectedRecord, unsigned int*, bool const&, yb::cdc::CDCSDKYsqlTest::ExpectedRecord, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool)
../../src/yb/integration-tests/cdcsdk_consistent_snapshot-test.cc:425:                                  @           0x425385  yb::cdc::CDCSDKConsistentSnapshotTest_InsertBeforeAfterSnapshot_Test::TestBody()
../../src/yb/integration-tests/cdcsdk_ysql_test_base.cc:1426: Failure
Expected equality of these values:
  expected_count[i]
    Which is: 1
  count[i]
    Which is: 2
Test failure stack trace:
../../src/yb/integration-tests/cdcsdk_ysql_test_base.cc:1426:                                           @     0x7f21bde3828c  yb::cdc::CDCSDKYsqlTest::CheckCount(unsigned int const*, unsigned int*)
../../src/yb/integration-tests/cdcsdk_consistent_snapshot-test.cc:427:                                  @           0x425288  yb::cdc::CDCSDKConsistentSnapshotTest_InsertBeforeAfterSnapshot_Test::TestBody()

or (saw in master)

../../src/yb/integration-tests/cdcsdk_consistent_snapshot-test.cc:417: Failure
Failed
Bad status: Not found (yb/cdc/cdc_service.cc:1825): Not leader for 4c405eb25b8d465ebd7b0a3d1a6411b6
[m-1] I0625 13:13:21.489457 4039813 async_rpc_tasks.cc:1556] Async Leader Stepdown (task=0x0000053cbf827690, state=kComplete): Leader step down done attempt=1, leader_uuid=4c52c81ba2354e6792a3bb8f9b6e2aaa, change_uuid=4c52c81ba2354e6792a3bb8f9b6e2aaa, error=, failed=0, should_remove=0 for tablet 4c405eb25b8d465ebd7b0a3d1a6411b6.
[ts-2] I0625 13:13:21.494902 4039519 tablet_service.cc:2620] vlog1: Received Run Leader Election RPC: tablet_id: "4c405eb25b8d465ebd7b0a3d1a6411b6"
dest_uuid: "e14040524ead4711b9c7e447f72d71be"
originator_uuid: "4c52c81ba2354e6792a3bb8f9b6e2aaa"

or (saw in master)

../../src/yb/integration-tests/cdcsdk_consistent_snapshot-test.cc:405: Failure                                                                                                                                                                                                                                                                                               Failed                                                                                                                                                                                                                                                                                                                                                                       Bad status: Illegal state (yb/consensus/replica_state.cc:1392): Not a leader

or (saw in pg15)

../../src/yb/integration-tests/cdcsdk_consistent_snapshot-test.cc:405: Failure
Failed
Bad status: Leader not ready to serve requests (yb/cdc/cdc_service.cc:1524): Not ready to serve

Source connector version

?

Connector configuration

?

YugabyteDB version

7217b58

Issue Type

kind/failing-test

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.