About Turo
Turo operates a peer-to-peer car-sharing platform that lets hosts share their vehicles with users. This innovative model serves as an alternative to traditional car rental services, enabling car owners to generate income from their vehicles when not in use.
Available in several countries, Turo is the world’s largest car-sharing marketplace. As of September 30, 2023, Turo boasts approximately 3.4 million active guests and 350,000 active vehicles across more than 12,000 cities in the US, UK, France, Canada, and Australia.
Permissions: stuck in park
As Turo's platform has evolved, its user base has rapidly expanded, with many hosts scaling their operations from one or two vehicles to managing large fleets. To assist with managing these fleets, hosts often shared login credentials with their employees, leading to significant security and usability issues. This revealed a critical need for a more secure and granular method of managing user access on the platform.
In response, Turo proposed a new co-hosting feature, enabling collaborative access and management of a single host account by multiple users. Development called for a flexible authorization system that could manage delegated vehicle access within a host account while supporting dual-role users (guests and hosts).
Unfortunately, their platform's existing authorization could not support the co-hosting requirements. As a result, the Turo team was faced with a tough decision: to refactor or not to refactor? Replacing the permissions system would require completely redefining user management and extensive rewrites across the platform's entire codebase. Not to mention, this change wouldn’t just be a challenge for the Co-hosting team, but it would be a blocker for other feature teams as well.
Ultimately, Turo made the strategic decision to refactor their authorization system. This decision meant more than just addressing a technical obstacle; it became a fundamental shift to empower their users for success. The Turo team aimed to significantly enhance the user experience and operational efficiency of the platform, underscoring their mission “to put the world’s 1.5 billion cars to better use”.
Journey to Zanzibar: researching a new authorization system
On the journey to refactoring a new authorization system, the API services team, led by Adam Safran, with Andre Sanches acting as technical lead for this project, began the discovery phase of this initiative with a destination in mind: Zanzibar.
Andre's previous research, originally conducted for a different project, identified Google Zanzibar as a promising standalone authorization model. This assessment was reinforced by Google's extensive utilization of Zanzibar and continued validation over time. Consequently, the team became intrigued by its potential relevance to Turo and was particularly interested in the model when it was publicly introduced in Google's 2019 paper.
Simultaneously, the team investigated authorization solutions employed by companies that shared similar growth trajectories and business models with Turo. This research brought them to Airbnb’s Himeji, a system inspired by Google Zanzibar, providing additional confirmation of the feasibility of a Zanzibar-based solution.
With the confidence of Zanzibar's success, the Turo team was determined to adopt a Zanzibar-based system. Nevertheless, because Himeji and Google Zanzibar were proprietary software, Turo needed an alternative implementation. After evaluating various open-source options and consulting with other organizations and technologists, they ultimately opted for SpiceDB.
Kicking the tires: adopting SpiceDB
The Turo team considered several factors when selecting their chosen system. The team wished to prevent vendor or ecosystem lock-in, which led them to reject other Zanzibar implementations that were part of larger network offerings. AuthZed’s complete focus on building authorization solutions vastly reduced this risk of vendor lock-in. This ensured Turo’s confidence in building their system, knowing they wouldn’t be obligated to integrate with a specific identity provider or other software for which they already had a solution.
To support organization-wide adoption, their solution needed to come equipped with robust documentation and developer tooling.
As Andre stated:
“What brought me over to SpiceDB was the documentation and the playground. When you’re introducing something that’s, in hindsight, so simple, yet appears complex when you first come across it, being able to show how it works in real-time with the playground is genius.”
After investigating the SpiceDB code, the Turo team found that the engineering maturity of SpiceDB:
- alleviated their concerns about integrating with legacy systems
- supported new functionality when scope-creep snuck into development
- and robustly handled gradual or complete feature rollouts
In a remarkable display of cross-functional collaboration, the API Services and Co-hosting feature teams joined forces to build a proof of concept with SpiceDB. Their collaborative PoC demonstration effectively rallied the engineering guild and Turo was off to the races!
Revving up: implementing SpiceDB at Turo
To ease application development, Turo developed a system to provision ephemeral SpiceDB environments. During peak usage, hundreds of developer systems run concurrently. To cut infrastructure costs, the system uses single SpiceDB deployment and relies on optional prefixes in SpiceDB for environment isolation.
Permissions data is critical to many stakeholders, not just those on the Application or API Services teams. As a result, Turo leverages Fine-Grained Access Management to restrict their access to the SpiceDB API so that they can only access relevant data and are unable to perform any write operations.
Like many large projects, requirements changed during the development process. When it was necessary to make design changes, Adam and Andre appreciated how simple it was to modify their SpiceDB schema. Often, they’d only need to alter a single line of code in their schema without changing any code in their applications.
Andre summarized:
"Here’s authorization. You don’t need to build anything else — you just need to talk to it."
Cruise control: SpiceDB in production
Because co-hosting relied on an invitation system, Turo lacked direct control over the feature’s user adoption once it was launched. To implement a gradual rollout, they opted to enable the feature one country at a time.
As we all know, sometimes, things don’t go according to plan! When it came time to deploy to the first country, the team accidentally deployed the functionality globally and the new system received 8x of the expected traffic.
Fortunately, SpiceDB performed without a hitch. As it turned out, the larger-than-planned roll-out had a silver lining: the Permissions System proved resilient to large-scale throughput effectively, dispelling any remaining doubts about SpiceDB's capability to handle critical use cases.
Andre recalls his takeaway from the experience:
"We have an API used by every single customer on every app in the Turo platform. I had a rough idea of the level of traffic, but I didn't know exactly what it was or how we'd ramp up to that traffic. The fact that SpiceDB took in that intense traffic without any problems made me feel a lot easier with my decision to use this technology."
AuthZed Dedicated: authZ in the fast lane
When teams contemplate adopting new software, they face a choice between two operational strategies: building or buying. Although developing their own Zanzibar implementation from scratch was never feasible within the timeline, the Turo team briefly considered operating SpiceDB themselves. However, after a quick discussion with the AuthZed customer success team, they determined that building and supporting the infrastructure themselves would not be cost-effective compared to purchasing AuthZed's fully managed and hosted product.
By opting for AuthZed Dedicated, Turo avoided the need to hire more engineers or overburden their existing team with tasks such as training, designing a production environment, deploying SpiceDB, and long-term infrastructure maintenance. These activities would have required extra budget allocation and, more importantly, would have diverted valuable engineering resources away from core product development.
Adam was eager to share his thought process when making this consideration:
"Authorization is something that has to be able to not only scale exceptionally, but also it's a decision on who is going to support that system. While we support [internal] advocacy, training, and consulting, one of the beautiful things about SpiceDB is that we're using the SaaS option so that we're not carrying the pagers for the authorization itself."
With their subscription to AuthZed Dedicated, Turo also gains access to AuthZed's Silver Support plan, which includes regular meetings with the customer success team and proactive support. During the onboarding process, Turo and AuthZed conducted architectural and schema reviews to align their system with design requirements, while also collaborating on optimizing SpiceDB to achieve Turo’s performance goals.
“The customer success team is fantastic. I have only praise to give them… They've always been very responsive.“
— Andre Sanches
Under the hood
Turo sends tens of millions of unique requests daily in SpiceDB, with an error rate of roughly one failed request per day, caused by unrelated issues. On the day the following results were collected, they handled 71.9 million requests, peaking at over 1,500 requests per second.
Andre is delighted by the results:
"If you look at the latencies, [...] the 99th percentile are all under 15 milliseconds in the worst case. Everything is just very, very impressive from anywhere you look at it. The vast majority of our calls are all within 3ms for check permission.
Everything looks just fantastic.”
