New: 2 dev tools launched! SpiceDB LSP support and VS Code extension now available

SpiceDB Tracing: How to interpret our OpenTelemetry traces

November 21, 2023|4 min read


This is Victor from AuthZed speaking and today we're going to look into some of the recent changes introducing the OTel instrumentation in SpiceDB.

Before these changes, folks needed a bit of knowledge on the internals of SpiceDB in order to interpret what was happening in requests, right?

It should be possible to visualize how SpiceDB walks the graph and that represents your schema. So you can better understand the sequence of steps taken to respond to an API request.

We're going to look into an example CheckPermission request using a simple schema with user and resource and the usual "view" permission, as you can see here which is granted if either user has a "reader" or "edit" relationship over a resource.

I have my SpiceDB instance running in the background with a Postgres datastore and I've spun up an instance of Jaeger, a popular distributed tracing service, and I've also just issued a check request with SpiceDB companion CLI tool, Zed. Let's have a look at the trace.

As you all probably know SpiceDB has a tunable consistency model which allows clients to decide the levels of consistency of their request.

By default, Zed uses minimize_latency which offers a good compromise between performance and security with a bounded staleness of up to five seconds maximum by default.

OptimizedRevision, ayou can see here at the top, is what indicates that minimized latency has been used as the server computes and a snapshot of the system and then SpiceDB validates the request parameters and making sure that it adheres to the schema that is currently in the system. As you can see in these two checks on the name spaces or the definitions on your schema.

Here's where the interesting bits start- this is at the first dispatch or we usually refer as sub problems being computed and we've issued a "view" over a specific resource for a user and so this is the first dispatch, this is the first sub-problem being computed.

This subproblem then SpiceDB determines that is composed of two parts or two sub problems: the "reader" side and the "edit" side. In turn, SpiceDB is going to turn these two guys into a new dispatch.

This is happening in one single instance so it's not actually dispatching to a different physical node, but for all intents and purposes, it should be the same. Here we can see then the subproblems of the first "view" permission check being computed.

We have "reader" on one side and then "edit" on the other side and the plus sign indicating this is a union. A "view" is a permission is granted to a user of resource over a resource if either they have "reader" or "edit" relationships.

When this is dispatched, then SpiceDB determines that this is a terminal or indicates that this is a relationship and the only thing it has to do is to actually query the database.

As you can see here it accesses the QueryRelationships datastore API and then issues a query "select." You can see details here of the actual SQL query issued to the Postgres database in case you're interested to know what's happening under the hood.

This is the new structure of the Otel traces that were introduced in SpiceDB 1.27.

Additional Reading

If you’re interested in learning more about Authorization and Google Zanzibar, we recommend reading the following posts:

Get started for free

Join 1000s of companies doing authorization the right way.