Skip to content

ape crash when i set network card response delay and run cdc sync #366

Open
@yester354

Description

@yester354

Bug description

  • Would you like to work on a fix? [y/n]

To Reproduce

Steps to reproduce the behavior:
mysql to mysql

  1. my target database
    sudo tc qdisc add dev eth0 root netem delay 35ms
    sudo tc qdisc add dev eth1 root netem delay 35ms.
  2. run snapshot sync
  3. .run cdc sync
  4. error as follow
    2025-04-21T10:11:54.064548+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.074645600+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.084706899+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.094759846+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.104815226+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.114870247+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.124959286+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.135028720+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.145083913+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.155130978+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.165192268+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.175251025+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.185318269+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.195375798+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.205425902+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.215484086+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.225538351+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.235618112+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.245688020+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.255741794+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.265795938+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.275852203+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.285916724+08:00 WARN mysql_binlog_connector_rust::network::packet_channel - Stream reading binlog returns zero-length data, Expected data length: 270, read so far: 242
    2025-04-21T10:11:54.291175382+08:00 INFO monitor_logger - extractor | | data_bytes | avg_by_sec=0 | sum=0 | max_by_sec=0
    2025-04-21T10:11:54.291190899+08:00 INFO monitor_logger - extractor | | record_count | avg_by_sec=0 | sum=0 | max_by_sec=0
    2025-04-21T10:11:54.291201717+08:00 INFO monitor_logger - pipeline | | buffer_size | avg=0 | sum=0 | max=0
    2025-04-21T10:11:54.291210827+08:00 INFO monitor_logger - pipeline | | record_size | avg=0
    2025-04-21T10:11:54.291219616+08:00 INFO monitor_logger - pipeline | | sinked_count | latest=16466
    2025-04-21T10:11:54.291229268+08:00 INFO monitor_logger - sinker | | rt_per_query | avg=0 | sum=0 | max=0
    2025-04-21T10:11:54.291238170+08:00 INFO monitor_logger - sinker | | records_per_query | avg=0 | sum=0 | max=0
    2025-04-21T10:11:54.291247114+08:00 INFO monitor_logger - sinker | | data_bytes | avg_by_sec=0 | sum=0 | max_by_sec=0
    2025-04-21T10:11:54.291255866+08:00 INFO monitor_logger - sinker | | record_count | avg_by_sec=0 | sum=0 | max_by_sec=0
    2025-04-21T10:11:54.302116554+08:00 ERROR default_logger - panic: panicked at /project/dt-task/src/task_runner.rs:453:39:
    called Result::unwrap() on an Err value: unexpected binlog data: Too many zero-length reads. Expected data length: 270, read so far: 242

Stack backtrace:
0: anyhow::error::<impl core::convert::From for anyhow::Error>::from
1: <dt_connector::extractor::mysql::mysql_cdc_extractor::MysqlCdcExtractor as dt_connector::Extractor>::extract::{{closure}}
2: dt_task::task_runner::TaskRunner::start_single_task::{{closure}}::{{closure}}
3: tokio::runtime::task::core::Core<T,S>::poll
4: tokio::runtime::task::harness::Harness<T,S>::poll
5: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
6: tokio::runtime::scheduler::multi_thread::worker::Context::run
7: tokio::runtime::context::runtime::enter_runtime
8: tokio::runtime::scheduler::multi_thread::worker::run
9: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
10: tokio::runtime::task::core::Core<T,S>::poll
11: tokio::runtime::task::harness::Harness<T,S>::poll
12: tokio::runtime::blocking::pool::Inner::run
13: std::sys::backtrace::__rust_begin_short_backtrace
14: core::ops::function::FnOnce::call_once{{vtable.shim}}
15: std::sys::pal::unix::thread::Thread::new::thread_start
16:
17:
backtrace:
0: dt_task::task_runner::TaskRunner::start_task::{{closure}}::{{closure}}
1: std::panicking::rust_panic_with_hook
2: std::panicking::begin_panic_handler::{{closure}}
3: std::sys::backtrace::__rust_end_short_backtrace
4: rust_begin_unwind
5: core::panicking::panic_fmt
6: core::result::unwrap_failed
7: dt_task::task_runner::TaskRunner::start_single_task::{{closure}}::{{closure}}
8: tokio::runtime::task::core::Core<T,S>::poll
9: tokio::runtime::task::harness::Harness<T,S>::poll
10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
11: tokio::runtime::scheduler::multi_thread::worker::Context::run
12: tokio::runtime::context::runtime::enter_runtime
13: tokio::runtime::scheduler::multi_thread::worker::run
14: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
15: tokio::runtime::task::core::Core<T,S>::poll
16: tokio::runtime::task::harness::Harness<T,S>::poll
17: tokio::runtime::blocking::pool::Inner::run
18: std::sys::backtrace::__rust_begin_short_backtrace
19: core::ops::function::FnOnce::call_once{{vtable.shim}}
20: std::sys::pal::unix::thread::Thread::new::thread_start
21:
22:

2025-04-21T10:11:54.302657694+08:00 ERROR default_logger - panic: panicked at /project/dt-task/src/task_runner.rs:473:31:
called Result::unwrap() on an Err value: JoinError::Panic(Id(41), "called Result::unwrap() on an Err value: unexpected binlog data: Too many zero-length reads. Expected data length: 270, read so far: 242\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From for anyhow::Error>::from\n 1: <dt_connector::extractor::mysql::mysql_cdc_extractor::MysqlCdcExtractor as dt_connector::Extractor>::extract::{{closure}}\n 2: dt_task::task_runner::TaskRunner::start_single_task::{{closure}}::{{closure}}\n 3: tokio::runtime::task::core::Core<T,S>::poll\n 4: tokio::runtime::task::harness::Harness<T,S>::poll\n 5: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n 6: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 7: tokio::runtime::context::runtime::enter_runtime\n 8: tokio::runtime::scheduler::multi_thread::worker::run\n 9: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll\n 10: tokio::runtime::task::core::Core<T,S>::poll\n 11: tokio::runtime::task::harness::Harness<T,S>::poll\n 12: tokio::runtime::blocking::pool::Inner::run\n 13: std::sys::backtrace::__rust_begin_short_backtrace\n 14: core::ops::function::FnOnce::call_once{{vtable.shim}}\n 15: std::sys::pal::unix::thread::Thread::new::thread_start\n 16: \n 17: ", ...)
backtrace:
0: dt_task::task_runner::TaskRunner::start_task::{{closure}}::{{closure}}
1: std::panicking::rust_panic_with_hook
2: std::panicking::begin_panic_handler::{{closure}}
3: std::sys::backtrace::__rust_end_short_backtrace
4: rust_begin_unwind
5: core::panicking::panic_fmt
6: core::result::unwrap_failed
7: dt_task::task_runner::TaskRunner::start_single_task::{{closure}}
8: dt_main::main::{{closure}}
9: tokio::runtime::park::CachedParkThread::block_on
10: dt_main::main
11: std::sys::backtrace::__rust_begin_short_backtrace
12: std::rt::lang_start::{{closure}}
13: std::rt::lang_start_internal
14: main
15:
16: __libc_start_main
17: _start

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions