Today, we're open sourcing prom-authzed-proxy!
prom-authzed-proxy is a proxy for Prometheus that authorizes the request's Bearer Token with Authzed and enforces a label in a PromQL query.
Authzed is a database and service that stores, computes, and validates your application's permissions.
Developers create a schema that models their permissions requirements and use a client library, such as this one, to apply the schema to the database, insert data into the database, and query the data to efficiently check permissions in their applications.
Previously on...
Last week, I wrote about how we're maintaining a single observability stack for both internal and external usage.
For SaaS products to truly become Cloud Native, they can no longer be treated as a black-box -- even services you don't run yourself must also be made observable.
To that end, there's now a metrics tab for each Permissions System in the Authzed dashboard that exposes latency metrics. When a user visits that page, their browser makes Prometheus API requests to the very same Prometheus deployment that we use internally.
To secure these requests, we built prom-authzed-proxy. And best of all, there is nothing in prom-authzed-proxy that makes it specific to our usage: anybody can use it to secure Prometheus. The proxy was built to have absolutely no assumptions about the schema of the Permission System or the metrics in Prometheus!
Prior Art
Proxying queries to Prometheus is not a new concept. In fact, it is the recommended way that the Prometheus community prescribes to implement secure querying. prom-label-proxy is a Prometheus community project that enforces that a particular label is present within a Prometheus request. By running prom-label-proxy, one can guarantee that requests are filtered down to only results from their Permissions System. However, this proxy has no means of authorizing that the request has access to the Permissions System that will be enforced on the query.
This is where prom-authzed-proxy comes in: prom-authzed-proxy is configured at startup with almost all of the arguments required to perform an Authzed permissions check.
The remaining arguments are parsed off of each request to the proxy: the contents of the Bearer Token is used as the Subject ID and the value of query parameter specified in the proxy's --object-id-parameter flag is used as the Object ID.
If the permissions check fails, the proxy returns an HTTP 403, otherwise it processes the request with the same library that handles requests internally to prom-label-proxy.
Next Time on...
In the future, we'll break down how to scale Prometheus to handle the cardinality imposed by labeling all of our metrics with each Permissions System.
Keep your eyes peeled for more blog posts or subscribe to monthly product updates direct to your inbox.
Huge thanks to Frederic Branczyk and Bartłomiej Płotka who jump-started our understanding of prior art in the Prometheus ecosystem.
Additional Reading
If you’re interested in learning more about Authorization and Google Zanzibar, we recommend reading the following posts:
