Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[experiment] expose readable datasets as dataframes and arrow tables #1507

Draft
wants to merge 8 commits into
base: devel
Choose a base branch
from

Conversation

sh-rp
Copy link
Collaborator

@sh-rp sh-rp commented Jun 21, 2024

Description

As an alternative to the ibis integration, we are testing out wether we can create our own data reader with not too much effort that works across all destinations.

Unordered notes:

  • Add some kind of interface to the filesystem so the user has some control over which files are considered for a request
  • Find out if exposing dataframes from jsonl files should be discouraged as duckdb probably will not be able to only read the exact right places in the file as it can with parquet
  • Should we allow the user to select which columns they want (probably yes), allow ordering (also probably yes, but with docs on which types of ordering are performant) or allow joins (I'm on the fence, user interviews will solve this)

Copy link

netlify bot commented Jun 21, 2024

Deploy Preview for dlt-hub-docs canceled.

Name Link
🔨 Latest commit 20bf9ce
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/667c3e9d6a1f7f0008eea44a

@sh-rp sh-rp linked an issue Jun 21, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

access data after load load as dataframes with ibis
1 participant