[Improve][Connector-V2][StarRocks]StarRocks Sink connector support 2pc and eos #4752
Conversation
|
The StarRocks E2E closed at now, can you change code to open it? |
| public String parseHttpResponse(CloseableHttpResponse response, String requestType) | ||
| throws StarRocksConnectorException { | ||
| int code = response.getStatusLine().getStatusCode(); | ||
| if (307 == code) { |
There was a problem hiding this comment.
Can you change the response code number to same meaning enum?
There was a problem hiding this comment.
Can you change the response code number to same meaning enum?
Thanks, fixed it
|
|
||
| @Override | ||
| public void init() { | ||
| this.manager = |
There was a problem hiding this comment.
Are there have a reason for why have to create an new thread?
There was a problem hiding this comment.
Creating a new thread is used to trigger flush batch data to stream load within the interval specified in "flush_frequency_ms". Batch writes can be triggered multiple times within a single transaction
|
check ci |
|
This PR is a big change for StarRocks connector. Please do more test and add test case. Thanks! |
| @@ -30,6 +38,8 @@ The internal implementation of StarRocks sink connector is cached and imported b | |||
| | retry_backoff_multiplier_ms | int | no | - | | |||
| | max_retry_backoff_ms | int | no | - | | |||
| | enable_upsert_delete | boolean | no | false | | |||
| | enable-2pc | boolean | no | false | | |||
There was a problem hiding this comment.
| | enable-2pc | boolean | no | false | | |
| | exactly-once | boolean | no | false | |
There was a problem hiding this comment.
Thank you for your suggestion, fix it
|
Please add test case to verify exactly-once. You can refer to https://github.com/apache/incubator-seatunnel/blob/dev/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterFaultToleranceIT.java |
...cks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/HttpHelper.java
Outdated
Show resolved
Hide resolved
...cks/src/main/java/org/apache/seatunnel/connectors/seatunnel/starrocks/client/HttpHelper.java
Outdated
Show resolved
Hide resolved
Thanks, I'll give it a try |
|
@531651225 Can we proceed with this PR? There are some conflicts that need to be resolved, and we also need to add some tests. |
|
This is a very good PR, do you have any plans to continue it |
yes，i will go on to try it |
Hello, is this still going on? I've been using the Starrcoks component recently and need this feature, if you need help, I'd be happy to |
Purpose of this pull request
StarRocks Sink connector support 2pc and eos
Testing under large data volume
Produce 500 million pieces of data using FakeSource, write them to starRocks through the starRocks sink connector 2pc method, and verify
refer to #4747
Check list
New License Guide
release-note.