You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I did not set the flink.execution.checkpointing.interval parameter in the configuration file,
The checkpoint.intercal parameter is not set in the Flink configuration file.
At this time, writing to ElasticSearch will lose data.
Because ElasticSearch Sink uses the parameter maxBatchSize to submit in batches, it will process uncommitted data through prepareCommit
@Override
public Optional<ElasticsearchCommitInfo> prepareCommit() {
bulkEsWithRetry(this.esRestClient, this.requestEsList);
return Optional.empty();
}
@Override
public void write(SeaTunnelRow element) {
if (RowKind.UPDATE_BEFORE.equals(element.getRowKind())) {
return;
}
String indexRequestRow = seaTunnelRowSerializer.serializeRow(element);
requestEsList.add(indexRequestRow);
if (requestEsList.size() >= maxBatchSize) {
bulkEsWithRetry(this.esRestClient, this.requestEsList);
}
}
This may be because the default value of checkpoint.interval is not set in Flink STREAMING mode in the code.
Search before asking
What happened
Described in the documentation:
![image](https://private-user-images.githubusercontent.com/29418975/339669251-474cc691-f414-4e4c-a154-9241f862bcec.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MDE5ODUsIm5iZiI6MTcxOTYwMTY4NSwicGF0aCI6Ii8yOTQxODk3NS8zMzk2NjkyNTEtNDc0Y2M2OTEtZjQxNC00ZTRjLWExNTQtOTI0MWY4NjJiY2VjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjI4VDE5MDgwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM4Y2FlZTI5MjkxY2RiY2NkNDM1ZTk0ZmZkYjk3MmJkNmYwMDRjYTM5NTUzNWJkNTFiM2ZhOTdjNzVhNzk0YjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.E1TGG8C9Jiokyncw-4fwzv9HwhI6bOOukYzg4lUzxII)
I submit the Flink task to Yarn for execution through the following command
test.config
I did not set the
flink.execution.checkpointing.interval
parameter in the configuration file,The
checkpoint.intercal
parameter is not set in the Flink configuration file.At this time, writing to ElasticSearch will lose data.
Because ElasticSearch Sink uses the parameter maxBatchSize to submit in batches, it will process uncommitted data through prepareCommit
This may be because the default value of checkpoint.interval is not set in Flink STREAMING mode in the code.
![image](https://private-user-images.githubusercontent.com/29418975/339670807-8d589fec-0c96-4eab-862c-ef4aba5d38b4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MDE5ODUsIm5iZiI6MTcxOTYwMTY4NSwicGF0aCI6Ii8yOTQxODk3NS8zMzk2NzA4MDctOGQ1ODlmZWMtMGM5Ni00ZWFiLTg2MmMtZWY0YWJhNWQzOGI0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjI4VDE5MDgwNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAzMDI4NzZkYzQ4NGE3NWU3MzM1OThmMGZlNmZmYjVjNTVjZTJlZTQxNmI5MWJlYWViN2E1MTk0MWY3YjkyZWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FFxO9HXkkQiynOXHrj885zAeLkS03uDbvKfHbZECyC0)
If this is a problem, please assign me.
SeaTunnel Version
2.3.5
SeaTunnel Config
Error Exception
Zeta or Flink or Spark Version
No response
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: