Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sycamore importer: detect insufficient memory or unimportable files and abort/skip them #158

Open
eric-anderson opened this issue Nov 9, 2023 · 0 comments

Comments

@eric-anderson
Copy link
Collaborator

The current importer will just keep retrying (with exponential backoff) when an import attempt fails. However, if the problem is insufficient memory or a document that causes the importer to crash, this is not a useful strategy.

Most likely after a few attempts to import in parallel fail, the importer should start importing documents serially and then mark ones that fail to import a few times as permanent failures and skip them until the importer code or document are updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant