Avoiding inconsistent relationships in the presence of concurrent Spice writes

[theronic]

"Last write wins" conditions during concurrent writes can result in stale Spice relationship that lead to persistent unwanted access — this is not a bug in Spice, just a reality of distributed systems with concurrent writers, even when the source of truth is consistent.

Our application is written in Clojure and uses Datomic as a source of truth. Datomic transactions cannot be wrapped around Spice writes, so without per-entity write locks or Spice Write Preconditions, this can lead to "last write wins" conditions. This affects daily operation but is most pronounced during long-running migrations.

I simulated concurrent writes to Datomic & Spice and can easily produce inconsistent state due to these scenarios. I was able to resolve all inconsistencies by adding an optimistic locking mechanism for each domain entity: I added a:spice/version counter to Datomic entities, which is incremented via CAS (compare-and-swap) after writing relationships to Spice. The compare-and-swap detects any concurrent writes and then retries syncing, resulting in very short-lived inconsistent state. Our workload is read-intensive, so this seems fine.

How do others deal with this? Write Preconditions are complicated for the caller to manage, because which preconditions do you check, especially during bulk updates? I have considered abusing a Spice relation as a lock (e.g. version relation that gets incremented).

Concrete example:

; Process A: add :owner rights to user-id
@(d/transact conn [[:db/add account-id :account/owner user-id]])
(spice/write-relationship! client (->user user-id) :owner (->account account-id))

; Process B: revoke :owner rights for user-id
@(d/transact conn [[:db/retract account-id :account/owner user-id]])
(spice/delete-relationship! client (->user user-id) :owner (->account account-id))

During concurrent execution, these commands can and do execute out-of-order and result in persistent unwanted user access:

@(d/transact conn [[:db/add account-id :account/owner user-id]]) ; process A
@(d/transact conn [[:db/retract account-id :account/owner user-id]]) ; process B
(spice/delete-relationship! client (->user user-id) :owner (->account account-id)) ; process A
(spice/write-relationship!  client (->user user-id) :owner (->account account-id)) ; process B
; ^ Result: unwanted access persists in Spice due to parallelism, even if Datomic commands run in-order.

Without write Preconditions, this can happen even if you compute the current relationships and issue bulk relationship updates (:delete & :touch), or if you delete relationships using RelationshipFilter.

Things get a bit more complicated during Spice schema & data migrations, where a migration could be long-running and span multiple Spice Commands. Even though Spice Commands are atomic, without write coordination, mid-migration writes can result in inconsistent Spice state. But at least we can query any entities that were written to during a migration and re-sync them when a migration completes.

Any advice on this would be appreciated.

[vroldanbet]

I have considered abusing a Spice relation as a lock (e.g. version relation that gets incremented).

@theronic the paper describes the use of a lock tuple to address this type of races, so it's not "abusing" the design at all:

https://zanzibar.tech/264hNtCdb3:0.BsRr6SyQa:1M

We've also discussed the notion of idempotency keys, which could help the adoption in event-driven architectures, feel free to chime in if it helps with your application: #1503

Things get a bit more complicated during Spice schema & data migrations, where a migration could be long-running and span multiple Spice Commands. Even though Spice Commands are atomic, without write coordination, mid-migration writes can result in inconsistent Spice state. But at least we can query any entities that were written to during a migration and re-sync them when a migration completes.

Ideally, you treat data migration like you do with relational databases, by running multiple phases. With the use of preconditions, data races should be accounted for:

• add new objects and relations to schema. It should be designed in a backward compatible way
• implement and deploy dual-writing to old and new relation. Do not update your permission yet, but you can add another permission if you want to test this in production with a subset of users.
• now it's time to run a backfill job. You want to select a SpiceDB snapshot revision (you can get this by doing a check or schema read), and do the migration at that snapshot revision. Please note that if the migration takes longer than the default 24h GC window, your snapshot reads will start to fail. You should estimate how long it would take and adjust GC window accordingly.
  • the key here to avoid races are the SpiceDB preconditions, every write you do should have a precondition that ensure the relationship in the original relation exists. If the precondition fails, you know a mutation happened since you did the snapshot read and thus would have been taken care for by the dual-write mechanism, so you'd skip that entry and continue with the next relationship.
• at this point the new relation is ready to go, and all you have to do is to update the schema to start using the new relation
• next step is to remove the dual-write in favour of only writing to the new relation
• finally you should run a delete job on the old relation, and then write the schema that drops the old relation