Keeping infrastructure software up to date has traditionally been a manual process: it was often part of somone's job description to check whether a new version of the software was available, verify its compatibility and deploy the update. However, this process has an Achilles heel: if the person (or people) responsible for checking for updates fails to do so, older versions of the software can languish.
Given the importance and benefits of keeping the software supply chain up to date, the Authzed team embarked on a small project recently to make it absolutely clear when updates for SpiceDB are available.
This blog post briefly discusses how SpiceDB and zed check for new versions and display that state to the end-user. The code used here is generally useful to any Go program that needs similar functionality.
The release process
SpiceDB is Authzed's open source, Zanzibar inspired database for fine-grained permissions. Releases of SpiceDB are found on GitHub, with built images automatically pushed to various container registries.
Using GitHub to publish releases has made the process for SpiceDB nearly seamless. The Authzed team quickly realized, however, that it left one major item to be desired: notification. While we did discover that GitHub provides an atom feed of available versions, the technology is no longer widely used, and as far as we knew, there was no way to easily indicate (outside of notifications in the SpiceDB Discord) to users of SpiceDB that new versions were available.
As new versions of SpiceDB can result in major performance improvements, new features, and even fix security issues, we felt it was important to make the availability of new versions as transparent to end users as possible.
Thus we embarked on a project to make both SpiceDB and zed display to the end user if the currently running version of SpiceDB can be updated.
Looking up releases
The first step in this process was to allow SpiceDB or zed to determine the latest version available.
Fortunately for us, the GitHub API provides a releases endpoint which lists all available releases for a project, as well as providing an endpoint returning just the latest release.
Even more fortunate was the discovery of a golang GitHub API client, which allowed for extremely easy interaction with the GitHub API.
Getting the latest released version of SpiceDB, therefore, became a simple GetLatestRelease function call:
client := github.NewClient(nil)
release, _, err := client.Repositories.GetLatestRelease(ctx, "authzed", "spicedb")
However, there was one issue with this endpoint: the latest release endpoint returns the last published non-prerelease release. If SpiceDB publishes a point release of an older version (say to backfill a bug fix), the older version will be reported as the latest released version! To counter this issue, we have instituted a policy to publish newer versions of the latest release in this scenario to ensure the real latest release is always returned. Suggestion to the GitHub API team: add an ability to sort by semantic version as well!
Getting the version of the binary
Once we have the latest released version, the next step was to retrieve the binary version for comparison.
Fortunately, we were again lucky that most of the infrastructure we needed had already been built: by setting the version into a build flag in Go, we were able to retrieve it at runtime. Furthermore, Go 1.18 provides the ability to access the git SHA used to build the binary automatically, which we were able to use as a fallback.
All of the above abilities were packaged into the cobrautil, the package SpiceDB uses to reduce CLI boilerplate package, so retrieving the version of the binary was similarly easy to do:
import (
"runtime/debug"
"github.com/jzelinskie/cobrautil"
)
// CurrentVersion returns the current version of the binary.
func CurrentVersion() (string, error) {
bi, ok := debug.ReadBuildInfo()
if !ok {
return "", fmt.Errorf("failed to read BuildInfo because the program was compiled with Go %s", runtime.Version())
}
return cobrautil.VersionWithFallbacks(bi), nil
}
Comparing versions
With the ability to retrieve the latest version, and the current version, the next step was to define a means for SpiceDB to compare versions. Fortunately, SpiceDB's versions (roughly) correspond to the use of semver, making the comparison between versions simple.
We were able, therefore, to define a CheckIsLatestVersion function for comparing versions of SpiceDB:
func CheckIsLatestVersion(
ctx context.Context,
getCurrentVersion func() (string, error),
getLatestRelease func(context.Context) (*Release, error),
) (SoftwareUpdateState, string, *Release, error) {
currentVersion, err := getCurrentVersion()
if err != nil {
return Unknown, currentVersion, nil, err
}
if currentVersion == "" || !semver.IsValid(currentVersion) {
return UnreleasedVersion, currentVersion, nil, nil
}
release, err := getLatestRelease(ctx)
if err != nil {
return Unknown, currentVersion, nil, err
}
if !semver.IsValid(release.Version) {
return Unknown, currentVersion, nil, err
}
if semver.Compare(currentVersion, release.Version) < 0 {
return UpdateAvailable, currentVersion, release, nil
}
return UpToDate, currentVersion, release, nil
}
Note an essential detail of this function: it takes in function closures to retrieve the latest and current versions, allowing it to elide some calls when not necessary. It also provides an easy means of testing by faking those calls.
Reporting to the user in SpiceDB
SpiceDB could now be designed to emit whether the running version was behind the latest available version with the comparison and loading functions complete.
The check was added via the use of a new pre-run function CheckAndLogRunE which (unless disabled by the --skip-release-check flag) grabs the latest release version from GitHub and compares it to the current version, logging if the version is out of date, or the version being run is not a released version:
3:00PM WRN this version of SpiceDB is out of date. See: https://github.com/authzed/spicedb/releases/tag/v1.7.1 this-version=1.7.0 latest-released-version=1.7.1
Reporting to the user in zed
Adding version checking to zed presented a slightly larger challenge: zed could easily make use of the CheckIsLatestVersion function for comparison, as it was included in the published releases package.
Likewise, zed could also use GetLatestRelease to retrieve the latest version of SpiceDB.
However, zed could not use GetCurrentVersion, as that would return the current version of zed instead of SpiceDB.
Thus, to support this use case, we needed to add a way for SpiceDB to return the running version to the caller of its APIs.
Fortunately, SpiceDB's API is implemented via gRPC, which has the concept of middleware for injecting custom logic into both client and server operations.
To add support for returning the current version, a new server side middleware was added to SpiceDB:
import (
"context"
"github.com/authzed/authzed-go/pkg/requestmeta"
"github.com/authzed/authzed-go/pkg/responsemeta"
"github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors"
"github.com/rs/zerolog/log"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
"github.com/authzed/spicedb/pkg/releases"
)
type handleServerVersion struct {
isEnabled bool
}
func (r *handleServerVersion) ServerReporter(ctx context.Context, _ interceptors.CallMeta) (interceptors.Reporter, context.Context) {
if r.isEnabled {
if md, ok := metadata.FromIncomingContext(ctx); ok {
if _, isRequestingVersion := md[string(requestmeta.RequestServerVersion)]; isRequestingVersion {
version, err := releases.CurrentVersion()
if err != nil {
log.Ctx(ctx).Err(err).Msg("could not load current software version")
return interceptors.NoopReporter{}, ctx
}
err = responsemeta.SetResponseHeaderMetadata(ctx, map[responsemeta.ResponseMetadataHeaderKey]string{
responsemeta.ServerVersion: version,
})
if err != nil {
log.Ctx(ctx).Err(err).Msg("could not report metadata")
}
}
}
}
return interceptors.NoopReporter{}, ctx
}
// UnaryServerInterceptor returns a new interceptor which handles server version requests.
func UnaryServerInterceptor(isEnabled bool) grpc.UnaryServerInterceptor {
return interceptors.UnaryServerInterceptor(&handleServerVersion{isEnabled})
}
// StreamServerInterceptor returns a new interceptor which handles server version requests.
func StreamServerInterceptor(isEnabled bool) grpc.StreamServerInterceptor {
return interceptors.StreamServerInterceptor(&handleServerVersion{isEnabled})
}
This middleware checks a specialized header on incoming requests, and, if present (and the middleware is not disabled via a flag), returns the server's current version in a gRPC response header.
Likewise, to perform the check on the zed side, a new client side middleware was used:
import (
"context"
"time"
"github.com/authzed/authzed-go/pkg/requestmeta"
"github.com/authzed/authzed-go/pkg/responsemeta"
"github.com/rs/zerolog/log"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
"github.com/authzed/spicedb/pkg/releases"
)
func CheckServerVersion(
ctx context.Context,
method string,
req, reply interface{},
cc *grpc.ClientConn,
invoker grpc.UnaryInvoker,
callOpts ...grpc.CallOption,
) error {
var headerMD metadata.MD
ctx = requestmeta.AddRequestHeaders(ctx, requestmeta.RequestServerVersion)
err := invoker(ctx, method, req, reply, cc, append(callOpts, grpc.Header(&headerMD))...)
if err != nil {
return err
}
version := headerMD.Get(string(responsemeta.ServerVersion))
if len(version) == 0 {
log.Debug().Msg("error reading server version response header; it may be disabled on the server")
} else if len(version) == 1 {
currentVersion := version[0]
rctx, cancel := context.WithTimeout(context.Background(), time.Second*2)
defer cancel()
state, _, release, cerr := releases.CheckIsLatestVersion(rctx, func() (string, error) {
return currentVersion, nil
}, releases.GetLatestRelease)
if cerr != nil {
log.Debug().Err(cerr).Msg("error looking up currently released version")
} else {
switch state {
case releases.UpdateAvailable:
log.Warn().Str("this-version", currentVersion).Str("latest-released-version", release.Version).Msgf("the version of SpiceDB being called is out of date. See: %s", release.ViewURL)
return nil
// … more cases handled here …
}
}
}
return err
}
The client side middleware sends the header to the server and, if it receives back a response with the server's version, logs the result of a call CheckIsLatestVersion.
Final thoughts
Adding release version checking and notification to SpiceDB and zed was a reasonably straightforward engineering project, but important design decisions allowed the library to be reusable, both for zed and others.
However, automatic release notification is only the first step to automating the running of SpiceDB.
Sign up for our mailing list to receive monthly SpiceDB and Authzed product updates and keep your eyes on this blog for more information about new developments!
Want to be notified directly when new releases of SpiceDB are available? Join the SpiceDB Discord to receive a direct notification whenever a new release is made!
Additional Reading
If you’re interested in learning more about Authorization and Google Zanzibar, we recommend reading the following posts:
