Skip to main content

Selecting a Datastore

SpiceDB ships with a number of datastores -- drivers used to store and retrieve the schema and relationship data for computing permissions.

There are a few available datastores with various design goals:

  • CockroachDB - Recommended for multi-region deployments
  • Cloud Spanner - Suitable for cloud multi-region deployments
  • MySQL - Recommended for single-region deployments and those familiar with traditional RDBMS operations
  • PostgreSQL - Recommended for single-region deployments and those familiar with traditional RDBMS operations
  • memdb - Recommended for local development and integration testing with applications written to be SpiceDB-native

Migrations

Before a datastore can be used by SpiceDB or before running a new version of SpiceDB, you must execute all available migrations. The only exception is the memdb datastore because it does not persist any data.

In order to migrate a datastore, run the following command with your desired values:

spicedb migrate head --datastore-engine $DESIRED_ENGINE --datastore-conn-uri $CONNECTION_STRING

CockroachDB

note

If you want to use the Watch API with CockroachDB, an additional capability must be enabled within the database. See Enabling Watch API for CRDB

Usage Notes

  • Recommended for multi-region deployments, with configurable region awareness
  • Enables horizonally scalablity by adding more SpiceDB and CockroachDB instances
  • Resiliency to individual CockroachDB instance failures
  • Query and data balanced across the CockroachDB
  • Setup and operational complexity of running CockroachDB

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using pgx for a SQL driver and connection pooling
  • Has a native changefeed
  • Stores migration revisions using the same strategy as Alembic

Configuration

danger

In distributed systems, you can trade-off consistency for performance.

Users that are willing to rely on subtle guarantees to mitigate the New Enemy Problem can configure --datastore-tx-overlap-strategy.

The available strategies are:

  • static (default) - A single key (--datastore-tx-overlap-key) is used in all writes to ensure proper consistency. This has the least write throughput, but is simplest and safest.
  • prefix (unsafe if misused) - A key with the prefix from the object type is used to protect writes with the same prefix. For example, subsystemA/user would overlap with subsystemA/document but not subsystemB/document.
  • request (unsafe if misused) - The user provides an overlap key per request as a gRPC request header. Only requests with the same overlap key will be protected. This allows an application to secure certain workflows, while getting full write performance elsewhere.
  • insecure (unsafe) - Disables the overlap strategy entirely, leaving queries vulnerable to the New Enemy problem, but provides the best write throughput possible on CockroachDB. Our docs on insecure overlap can help determine if the trade-offs of insecure are right for you.

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=cockroachdb
datastore-conn-uriconnection string used to connect to CRDB--datastore-conn-uri="postgres://user:password@localhost:26257/spicedb?sslmode=disable"

Optional Parameters

ParameterDescriptionExample
datastore-max-tx-retriesMaximum number of times to retry a query before raising an error--datastore-max-tx-retries=50
datastore-tx-overlap-strategyThe overlap strategy to prevent New Enemy on CRDB (see below)--datastore-tx-overlap-strategy=static
datastore-tx-overlap-keyThe key to use for the overlap strategy (see below)--datastore-tx-overlap-key="foo"
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-conn-min-openMinimum number of concurrent connections to open--datastore-conn-min-open=1
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true
datastore-follower-read-delay-durationAmount of time to subtract from non-sync revision timestamps to ensure follower reads--datastore-follower-read-delay-duration=4.8s

Garbage Collection Window

Why did I get a warning about the garbage collection window size?

Cockroach DB has (as of Feb 2023) recently changed the default garbage collection window size to 1.5 hours for CRDB Serverless and 4 hours for CRDB Dedicated.

SpiceDB will read and use the setting from CRDB, but reports a warning if the (SpiceDB) configured window is larger in size.

If you need a longer time window for Watch or Snapshots, please adjust on the CRDB side:

ALTER ZONE default CONFIGURE ZONE USING gc.ttlseconds = 90000;

Cloud Spanner

Usage Notes

  • Requires a Google Cloud Account with an active Cloud Spanner instance
  • Take advantage of Google's TrueTime. The Spanner driver assumes the database is linearizable and skips the transaction overlap strategy required by CockroachDB.

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Starts a background GC worker to clean up old entries from the manually-generated changelog table

Configuration

  • The Cloud Spanner docs outline how to set up an instance
  • Authentication via service accounts: The service account that runs migrations must have Cloud Spanner Database Admin; SpiceDB (non-migrations) must have Cloud Spanner Database User.

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=spanner
datastore-conn-urithe cloud spanner database identifier--datastore-conn-uri="projects/project-id/instances/instance-id/databases/database-id"

Optional Parameters

ParameterDescriptionExample
datastore-spanner-credentialsJSON service account token (omit to use application default credentials)--datastore-spanner-credentials=./spanner.json
datastore-gc-intervalAmount of time to wait between garbage collection passes--datastore-gc-interval=3m
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true
datastore-follower-read-delay-durationAmount of time to subtract from non-sync revision timestamps to ensure stale reads--datastore-follower-read-delay-duration=4.8s

MySQL

Usage Notes

  • Recommended for single-region deployments
  • Setup and operational complexity of running MySQL
  • Does not rely on any non-standard MySQL extensions
  • Compatible with managed MySQL services

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using Go-MySQL-Driver for a SQL driver
  • Query optimizations are documented here
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=mysql
datastore-conn-uriconnection string used to connect to MySQL--datastore-conn-uri="user:password@(localhost:3306)/spicedb?parseTime=True"
MySQL Time
note

--datastore-conn-uri must contain parseTime=true as a query parameter in order support time-based operations such as Garbage Collection in SpiceDB.

Example: --datastore-conn-uri="username:password@(localhost:3306)/spicedb?parseTime=True"

Optional Parameters

ParameterDescriptionExample
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-mysql-table-prefix stringPrefix to add to the name of all SpiceDB database tables--datastore-mysql-table-prefix=spicedb
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true

PostgreSQL

note

If you want to use the Watch API with PostgreSQL, an additional capability must be enabled within the database. See Enabling Watch API for PostgreSQL

note

The minimum supported PostgreSQL version is 13.8

Usage Notes

  • Recommended for single-region deployments
  • Resiliency to failures only when PostgreSQL is operating with a follower and proper failover
  • Setup and operational complexity of running PostgreSQL
  • Does not rely on any non-standard PostgreSQL extensions
  • Compatible with managed PostgreSQL services

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implemented using pgx for a SQL driver and connection pooling
  • Stores migration revisions using the same strategy as Alembic
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine=postgres
datastore-conn-uriconnection string used to connect to PostgreSQL--datastore-conn-uri="postgres://postgres:password@localhost:5432/spicedb?sslmode=disable"

Optional Parameters

ParameterDescriptionExample
datastore-conn-max-idletimeMaximum idle time for a connection before it is recycled--datastore-conn-max-idletime=60s
datastore-conn-max-lifetimeMaximum lifetime for a connection before it is recycled--datastore-conn-max-lifetime=300s
datastore-conn-max-openMaximum number of concurrent connections to open--datastore-conn-max-open=10
datastore-conn-min-openMinimum number of concurrent connections to open--datastore-conn-min-open=1
datastore-query-split-sizeThe (estimated) query size at which to split a query into multiple queries--datastore-query-split-size=5kb
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true

memdb

Usage Notes

  • Fully ephemeral; all data is lost when the process is terminated
  • Intended for usage with SpiceDB itself and testing application integrations
  • Cannot be ran highly-available as multiple instances will not share the same in-memory data
note

If you need an ephemeral datastore designed for validation or testing, see the test server system in Validating and Testing

Developer Notes

  • Code can be found here
  • Documentation can be found here
  • Implements its own MVCC model by storing its data with transaction IDs

Configuration

Required Parameters

ParameterDescriptionExample
datastore-enginethe datastore engine--datastore-engine memory

Optional Parameters

ParameterDescriptionExample
datastore-revision-fuzzing-durationSets a fuzzing window on all zookies/zedtokens--datastore-revision-fuzzing-duration=50ms
datastore-gc-windowSets the window outside of which overwritten relationships are no longer accessible--datastore-gc-window=1s
datastore-readonlyPlaces the datastore into readonly mode--datastore-readonly=true