You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now the importer has no way to notice that a crawled file has been deleted and remove it from the index. Similarly, if a file is updated, the crawlers will detect that, re-download the file and the importer will then import it again without removing the old file resulting in duplicate entries in the index.
It's unclear what the right way to handle this is, but a likely choice is that the files stored in the crawl_data compose volume should match the ones stored in opensearch and there should be no duplicates.
The text was updated successfully, but these errors were encountered:
Right now the importer has no way to notice that a crawled file has been deleted and remove it from the index. Similarly, if a file is updated, the crawlers will detect that, re-download the file and the importer will then import it again without removing the old file resulting in duplicate entries in the index.
It's unclear what the right way to handle this is, but a likely choice is that the files stored in the crawl_data compose volume should match the ones stored in opensearch and there should be no duplicates.
The text was updated successfully, but these errors were encountered: