New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ [tarchia] use a btree to store manifests #1730

Open

joocer opened this issue Jun 4, 2024 · 1 comment

Contributor

joocer commented Jun 4, 2024

Something like 20k entries per manifest, when we hit the limit split the manifest and create a root pointing to the split files.

Use tree balancing algorithms for things like btrees to balance the manifests.

Use the system date as the index column.

Contributor Author

joocer commented Jun 9, 2024 •

edited

Loading

Doing some testing, with a 2mb cache, it appears that as few as 1000 records would be the limit. In contrived testing 1000 records was 1.6mb.

Whilst this is a lot smaller than I was hoping, we can still store 1 million items in a single manifest and one billion in 67 manifests.

We would probably make it 1024 to make some math easier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment