The advertiser platform
Reddit, the social news site and eighth most visited website in the world, serves over 400 million monthly users in nearly 140,000 active communities (subreddits). Reddit Ads, the company’s advertising platform, empowers advertisers to reach those users. When the product team for the ads platform discovered that their authorization needs had outgrown an existing homegrown system, they opted for AuthZed’s open-source Zanzibar implementation SpiceDB to take on the challenge.
Sumedha Raman and Braden Groom are on the product team for Reddit Ads. They sat down with AuthZed to share their journey to SpiceDB in production.
Reddit Ads has the many capabilities you would expect from a mature ads offering:
- Advertisers can create ads, define and target audiences, and set creatives.
- Advertisers upload and define ad assets, the text and media served to target audiences.
- Dashboards and reporting provide advertisers with business metrics to help them understand the outcomes of their ad spending.
Some of Reddit’s largest advertising customers (like Amazon) require advanced features, such as dynamic ad creation, where advertisements are derived from product catalogs due to the volume of assets that need to be created. As Braden described, “They don’t want to create one-off ads, they want to upload a CSV with 5 million products and just dynamically create ads.”
This system is robust, with many features serving the many personas and businesses using it - and the need for authorization arises across all these features. The platform manages many types of assets, such as ad accounts, audiences, media collections (images, videos), and profiles (an identity on behalf of which ads are displayed). Businesses on the platform have employees (user accounts) who then access these assets in various ways and with varying authorization rules.
It’s possible for a user on the platform to have both direct and indirect access to these assets. For example, a user may have direct access to an ad account (if they are an admin on that ad account, say). Or, they may have indirect access, such as access to a profile belonging to an ad account, where the user only has been granted direct access to the ad account. In this way, they are granted access to the profile through their access to the associated ad account. Sumedha and Braden call these “transitive permissions.” But when businesses began collaborating, things got even more complicated.
Some bigger brands collaborate with other businesses by running ads through agencies, leading to even more complex authorization requirements.
“We're moving towards this world in which an ad is not created in isolation with a single business, but it may be the gathering of many assets from different businesses. That gets pretty complicated and was a major part of what made us realize we needed a different solution.”
Outgrowing the homegrown solution
“The need for a new solution arose for us in the summer of 2019, which was around the same time the Zanzibar paper came out,” noted Braden. There was no open-source Zanzibar implementation immediately available, so the product team decided to build an internal system inspired by the paper. With so many other priorities, though, they didn’t have free reign to spend a year building a full graph-based solution for authorization.
They built what they called a “sharded RBAC” system, which worked well, but it only brought them so far. The need for sharing assets and information across businesses, an important use case for big customers, triggered them to search for a new solution. The homegrown “sharded” solution didn’t allow for permissions to be defined across multiple businesses.
The homegrown system also had initial limits to protect for performance, such as 1000 permissions per business. “In 2019,” Braden said, “our top businesses had 20-30 permissions. As our product surface areas grew, we hit that 1000 limit quickly.” As larger companies joined the advertising platform, the 1000 permission limit was far too low.
SpiceDB: inspiring confidence in Zanzibar
The ads team did their research on the many available Zanzibar implementations, but they kept coming back to SpiceDB.
“SpiceDB inspired the most confidence. There was so much documentation. We were impressed by the faithfulness to the Zanzibar paper and by the team behind SpiceDB. It’s clear the team has thought deeply about the authorization space.”
Sumedha and Braden also noted that SpiceDB stood out for its strong presence in the developer ecosystem, robust community and support, and specific features they needed like bulk permission checks.
“We evaluated many other Zanzibar solutions, but it came down to features. Our other top contenders didn’t have the specific features we needed, including bulk permission checks, and SpiceDB did.”
Beyond bulk permission checks, SpiceDB was equipped for some of the more interesting problems the product team faced, including a graph reachability problem around who else has access to a resource. “We wanted to show users others who had access to something in common, which becomes a graph problem very quickly,” said Sumedha. “That was a complex implementation and it started getting slow, but it’s something SpiceDB solves for.”
SpiceDB’s flexibility was a major factor in the choice, especially when it came to complex asset sharing for agencies and businesses. “It's hard to just say we support sharing this one asset across and that solves the problem of businesses and agencies,” said Sumedha. “Every agency-business combination wants to set it up in a different way. It just needs to be flexible.”
The final decision seemed simple. A team with deep authorization experience, unique features in the authorization market, a flexible system, and the ability to resolve complex graph lookups – check, check, check, and check.
Putting authorization to work: implementing SpiceDB
At Reddit ads, the SpiceDB implementation was a big deal. “Ultimately, there was interest in SpiceDB becoming a bigger part of the Reddit platform,” noted Sumedha. “The infrastructure team wanted to be sure we were heading in a direction that could be useful to Reddit in general, so the implementation involved an audit of all the authorization use cases across Reddit, and that’s when some of the interesting problems like moderator permissions came up. We wanted to show teams across the company that SpiceDB would support their use case.”
The migration to SpiceDB took a community of people. The infrastructure and operations team helped to set up the SpiceDB operator, and the storage team helped set up the Postgres instance and the storage pieces. Sumedha and Braden’s team migrated the schema and worked through the code changes to use SpiceDB. “We repurposed our existing authorization API layer to make the migration simpler, so we only had to update authorization checks in one place,” explained Sumedha. “This also allowed us to log the results of every authorization check into a common event system.”
Their rollout employed a “tap and compare” phase: the team ran SpiceDB in parallel with the old authorization system, investigating and fixing any discrepancies as they arose. This strategy allowed the Reddit Ads team to thoroughly test and validate their SpiceDB implementation before fully transitioning to the new system. It gave teams a high degree of confidence in the ultimate success of the migration. Overall, the thoroughness of their rollout is representative of the sound engineering practices employed at Reddit Ads.
During this process, whenever discrepancies were encountered (between the results coming from the old system and SpiceDB) - most of the time it was “SpiceDB doing the right thing,” according to Sumedha. This resulted in the team working to ensure that these discrepancies were tracked and well understood so that any different results would make sense in the product after the cutover was finalized.
Additionally, the team ended up tracking more information in SpiceDB than in their previous authorization implementation. This created a situation where dual writes were required for a period, as the “tap and compare” ran. It also called for backfilling information to ensure that both systems could be compared properly with one another.
Reddit’s security team also played a role in validating the migration. They conducted a two-week testing period to re-test the entire platform, which yielded no critical issues. From Reddit security’s perspective, the migration looked good, confirming that no other parts of the system were impacted.
Ultimately, SpiceDB was only launched to 100% of requests once there were no discrepancies between the old and the new system, or any discrepancies that did occur made full sense to the team (and were validated to not cause product regressions). The goal was to ensure that the platform continued to work as expected and that users wouldn’t notice any differences.
Throughout this process, the team also encountered the trade-offs between consistency and latency firsthand. Everything ran with full consistency during the initial rollout. The team later adopted ZedTokens and additional consistency options to further improve latency, once the core correctness of the system was fully vetted and validated.
A successful launch and a Reddit-wide future
The Reddit ads team measured success simply: the platform would work exactly as it did before. “The idea was, if no one noticed, it would be successful,” Sumedha said. “And for the most part, no one did! We ran our ‘tap and compare’ until there were no inconsistencies between the old system and the new system.”
Reddit security audited the entire SpiceDB rollout and found no issues. This was key since the ultimate goal is to make SpiceDB a company-wide platform. “We want it to be a core piece of tech at Reddit, used by multiple teams,” said Braden. “Our next goal is to get moderator permissions to adopt SpiceDB. Ideally, we’ll get them to share a graph with Ads and get the benefits of a global graph.”
Adopting SpiceDB has also influenced the road map for the advertiser platform.
“The whole road map is now essentially different types of permissions and authorization features. That’s all possible now thanks to SpiceDB.”
