Importance of sorting when using multiple transaction files #503
Replies: 1 comment 1 reply
-
This is a good point and valid that order matters in a Stack data structure. If we can ensure that ordering is done either by file or within files, it will be better than having to do a sort on all transactions. I agree that it may be largely academic, but it is still best practice. @pjohnmeyer @JSv4 Can we discuss at the next TWG and decide a proposal. If we are enforcing ordering of some sort, then we would need to validate that ordering is being adhered to. Either through the OCF schema validator or in the validation tool itself. And it may be that everyone else is ordering and I am the only one assuming that a transaction file may or may not be ordered in which case we should validate ordering. |
Beta Was this translation helpful? Give feedback.
-
Background:
** Now I admit, the problem was largely theoretical. In practice would we ever have a cap table so large that the data can't actually fit within the memory constraints of our modern hardware?
** But regardless, theoretical or not, it was deemed important enough to make the change and introduce the ZIP structure.
Today:
Order Important?
The OCF Architecture is described as a stack architecture. It's inherent that order matters when using a Stack data structure. But are we generally in agreement that a given OCF implementation needn't worry about ordering the transactions within the transactions file(s)? It's a logical impediment to make assumptions.
Perhaps within a single transactions file, the order needn't matter. But in the OCF documentation (on the OCFManifestFile docs) we state a sentence like this for each of the arrays of file references:
Meaning OCF expected to be able to process the files one-by-one, and for that to be possible, we need sorting to be in logical order.
Opening this up for discussion as there seems to be some confusion on the topic. If order truly doesn't matter (and the size problem is indeed theoretical only) then we could return to a simplified OCF JSON format instead of the multi-file architecture.
Beta Was this translation helpful? Give feedback.
All reactions