You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I uploaded data from Kafka to S3 every 30 minutes.
However, when the amount of data increases, the last modified time is wrong.
Moreover, Last modified time was "earlier than" actual upload time.
For example below,my files at 21:23:54, 21:24:29 have all data from 21:00:00~21:29:59
There are no issues with actual upload time and data integrity.
The only problem is the "last-modified time indicated in S3".
I found one more.
This problem occurs when the size of file is bigger than about 26MiB. The "last modified"is recorded 1 minute faster for every about 1MiB increase from 26MiB.
My Network bandwidth and computing power are sufficient. I doubt it is due to lack of topic's partitions. But, Where is the Criteria for selecting the number of partitions in terms of file size or transfer speed?
Hi there,
Version
S3 Sink Connector: 10.5.0
Kafka: 3.5.0
Problem
I uploaded data from Kafka to S3 every 30 minutes.
However, when the amount of data increases, the last modified time is wrong.
Moreover, Last modified time was "earlier than" actual upload time.
For example below,my files at 21:23:54, 21:24:29 have all data from 21:00:00~21:29:59
There are no issues with actual upload time and data integrity.
The only problem is the "last-modified time indicated in S3".
Anybody who know this issue? Thank you.
The text was updated successfully, but these errors were encountered: