Add: Import and export tool for csv/parquet/json #181
Conversation
…tjson-arrow-datasets
…to 154-implement-importexport-tool-for-csvparquetjson-arrow-datasets
| void import_parquet(ukv_graph_import_t& c, ukv_size_t max_batch_size) { | ||
|
|
||
| arrow::Status status; | ||
| arrow::MemoryPool* pool = arrow::default_memory_pool(); |
There was a problem hiding this comment.
We can use our arenas, like in client
There was a problem hiding this comment.
Its not working with parquet.
tools/dataset.cpp
Outdated
|
|
||
| void import_json(ukv_graph_import_t& c, ukv_size_t max_batch_size) { | ||
|
|
||
| std::vector<edge_t> array; |
There was a problem hiding this comment.
cant we preallocate a max size vector?
There was a problem hiding this comment.
For small data, this may be overkill.
There was a problem hiding this comment.
These tools are only intended for large imports
fix:/refactor: minor
ashvardanian
changed the title
… always converted to true
src/modality_docs.cpp
Outdated
| @@ -1281,6 +1287,30 @@ void ukv_docs_write(ukv_docs_write_t* c_ptr) { | |||
| linked_memory_lock_t arena = linked_memory(c.arena, c.options, c.error); | |||
| return_on_error(c.error); | |||
|
|
|||
| std::vector<ukv_key_t> keys_vec; | |||
There was a problem hiding this comment.
No std::vector-s allowed... We have to use our own memory.
src/modality_docs.cpp
Outdated
| strided_iterator_gt<ukv_length_t const> lens {c.lengths, c.lengths_stride}; | ||
|
|
||
| for (size_t idx = 0; idx < c.tasks_count; ++idx, ++vals, ++lens) { | ||
| simdjson::ondemand::parser parser; |
tools/dataset.cpp
Outdated
| return arrow::Status(arrow::StatusCode::TypeError, "Not supported type"); | ||
| } | ||
| arrow::Status Visit(arrow::BooleanArray const& arr) { | ||
| json = fmt::format("{}{},", json, arr.Value(idx)); |
There was a problem hiding this comment.
This is horrible.
You are reallocating a new string every time you want to append a boolean.
At least use fmt::format_to.
tools/dataset.cpp
Outdated
|
|
||
| ///////// Helpers ///////// | ||
|
|
||
| class arrow_visitor { |
There was a problem hiding this comment.
Check styling guidelines.
tools/dataset.cpp
Outdated
| size_t idx = 0; | ||
| }; | ||
|
|
||
| bool strcmp_(const char* lhs, const char* rhs) { |
There was a problem hiding this comment.
Style guidelines: East const
| docs_vec.reserve(size); | ||
|
|
||
| if (c.fields) { | ||
| std::vector<std::string> fields(c.fields_count); |
There was a problem hiding this comment.
Why do you need a std::vector or std::string?
tools/dataset.cpp
Outdated
|
|
||
| char file_name[uuid_length]; | ||
| make_uuid(file_name); | ||
| std::ofstream output(fmt::format("{}{}", file_name, c.paths_extension)); |
There was a problem hiding this comment.
Please, never use ofstream or the rest of the old-school IO libraries from STL. Especially on the hot path.
…tjson-arrow-datasets
ashvardanian
deleted the
154-implement-importexport-tool-for-csvparquetjson-arrow-datasets
branch
No description provided.