Skip to main content

Selecting a Datastore

SpiceDB ships with a number of datastores -- drivers used to store and retrieve the schema and relationship data for computing permissions.

There are a few available datastores with various design goals:

  • CockroachDB - Recommended for multi-region deployments
  • Cloud Spanner - Suitable for cloud multi-region deployments
  • MySQL - Recommended for single-region deployments and those familiar with traditional RDBMS operations
  • PostgreSQL - Recommended for single-region deployments and those familiar with traditional RDBMS operations
  • memdb - Recommended for local development and integration testing with applications written to be SpiceDB-native

Migrations

Before a datastore can be used by SpiceDB or before running a new version of SpiceDB, you must execute all available migrations. The only exception is the memdb datastore because it does not persist any data.

In order to migrate a datastore, run the following command with your desired values:

spicedb migrate head --datastore-engine $DESIRED_ENGINE --datastore-conn-uri $CONNECTION_STRING

CockroachDB

note

If you want to use the Watch API with CockroachDB, an additional capability must be enabled within the database. See Enabling Watch API for CRDB

Usage Notes

  • Recommended for multi-region deployments, with configurable region awareness
  • Enables horizonally scalablity by adding more SpiceDB and CockroachDB instances
  • Resiliency to individual CockroachDB instance failures
  • Query and data balanced across the CockroachDB
  • Setup and operational complexity of running CockroachDB

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using pgx for a SQL driver and connection pooling
  • Has a native changefeed
  • Stores migration revisions using the same strategy as Alembic

Configuration

danger

In distributed systems, you can trade-off consistency for performance.

Users that are willing to rely on subtle guarantees to mitigate the New Enemy Problem can configure --datastore-tx-overlap-strategy.

The available strategies are:

  • static (default) - A single key (--datastore-tx-overlap-key) is used in all writes to ensure proper consistency
  • prefix (unsafe if misused) - A key with the prefix from the object type is used to protect writes with the same prefix
  • insecure (unsafe) - Disables the overlap strategy entirely leaving queries vulnerable to the New Enemy problem

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=cockroachdb
datastore-conn-uriconnection string used to connect to CRDB--datastore-conn-uri="postgres://user:password@localhost:26257/spicedb?sslmode=disable"

Optional Parameters

ParameterDescriptionExample
datastore-max-tx-retriesMaximum number of times to retry a query before raising an error--datastore-max-tx-retries=50
datastore-tx-overlap-strategyThe overlap strategy to prevent New Enemy on CRDB (see below)--datastore-tx-overlap-strategy=static
datastore-tx-overlap-keyThe key to use for the overlap strategy (see below)--datastore-tx-overlap-key="foo"
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-conn-min-openMinimum number of concurrent connections to open--datastore-conn-min-open=1
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true
datastore-follower-read-delay-durationAmount of time to subtract from non-sync revision timestamps to ensure follower reads-datastore-follower-read-delay-duration=4.8s

Cloud Spanner

Usage Notes

  • Requires a Google Cloud Account with an active Cloud Spanner instance
  • Take advantage of Google's TrueTime. The Spanner driver assumes the database is linearizable and skips the transaction overlap strategy required by CockroachDB.

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Starts a background GC worker to clean up old entries from the manually-generated changelog table

Configuration

  • The Cloud Spanner docs outline how to set up an instance
  • Authentication via service accounts: The service account that runs migrations must have Cloud Spanner Database Admin; SpiceDB (non-migrations) must have Cloud Spanner Database User.

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=spanner
datastore-conn-urithe cloud spanner database identifier--datastore-conn-uri="projects/project-id/instances/instance-id/databases/database-id"

Optional Parameters

ParameterDescriptionExample
datastore-spanner-credentialsJSON service account token (omit to use application default credentials)--datastore-spanner-credentials=./spanner.json
datastore-gc-intervalAmount of time to wait between garbage collection passes--datastore-gc-interval=3m
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true
datastore-follower-read-delay-durationAmount of time to subtract from non-sync revision timestamps to ensure stale reads--datastore-follower-read-delay-duration=4.8s

MySQL

Usage Notes

  • Recommended for single-region deployments
  • Setup and operational complexity of running MySQL
  • Does not rely on any non-standard MySQL extensions
  • Compatible with managed MySQL services

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using Go-MySQL-Driver for a SQL driver
  • Query optimizations are documented here
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=mysql
datastore-conn-uriconnection string used to connect to MySQL--datastore-conn-uri="mysql:password@localhost:5432/spicedb?parseTime=True&sslmode=disable"
MySQL Time
note

--datastore-conn-uri must contain parseTime=true as a query parameter in order support time-based operations such as Garbage Collection in SpiceDB.

Example: --datastore-conn-uri="mysql:password@localhost:5432/spicedb?sslmode=disable&parseTime=True"

Optional Parameters

ParameterDescriptionExample
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-mysql-table-prefix stringPrefix to add to the name of all SpiceDB database tables--datastore-mysql-table-prefix=spicedb
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true

PostgreSQL

note

If you want to use the Watch API with PostgreSQL, an additional capability must be enabled within the database. See Enabling Watch API for PostgreSQL

Usage Notes

  • Recommended for single-region deployments
  • Resiliency to failures only when PostgreSQL is operating with a follower and proper failover
  • Setup and operational complexity of running PostgreSQL
  • Does not rely on any non-standard PostgreSQL extensions
  • Compatible with managed PostgreSQL services

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using pgx for a SQL driver and connection pooling
  • Stores migration revisions using the same strategy as Alembic
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=postgres
datastore-conn-uriconnection string used to connect to PostgreSQL--datastore-conn-uri="postgres://postgres:password@localhost:5432/spicedb?sslmode=disable"

Optional Parameters

ParameterDescriptionExample
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-conn-min-openMinimum number of concurrent connections to open--datastore-conn-min-open=1
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true

memdb

Usage Notes

  • Fully ephemeral; all data is lost when the process is terminated
  • Intended for usage with SpiceDB itself and testing application integrations
  • Cannot be ran highly-available as multiple instances will not share the same in-memory data
note

If you need an ephemeral datastore designed for validation or testing, see the test server system in Validating and Testing

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine memory

Optional Parameters

ParameterDescriptionExample
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true