Matillion ensures success for the modern data team. They offer a comprehensive data pipeline platform that enables companies to move and transform their data, delivering analytics and AI at scale. Empowering data teams since 2011, Matillion’s cloud-first approach continuously evolves and incorporates innovative technologies. The latest evolution, Matillion Data Productivity Cloud, brings the power of Matillion’s self-managed ETL solutions to a comprehensive SaaS-based platform. Of course, accomplishing a transformation of this magnitude is no small task.
We sat down with members of Matillion’s engineering and product teams to learn more about this transition and what brought them to AuthZed.
Enterprise workflows, meet fine-grained permissions
With the launch of Matillion’s Data Productivity Cloud, the team sought to build an everyone-ready data platform, which meant designing workflows suited to many additional personas that could be eventually further customized by end users. These workflows included everything from providing billing departments access to subscription management to defining environment-level permissions for data analysts.
Building collaborative workflows relies on a permissions system to set boundaries on users’ activities by managing what they can access, whether it’s a particular resource or a feature of the product. The Matillion team recognized the need for three levels of access management across the platform:
- Account-level: administrative and application access
- Project-level: workspace access
- Environment-level: pipeline management access
As Matillion designed more sophisticated permissions workflows with finer granularity and multiple layers of permissions nesting, their initial JWT-based approach lacked the flexibility and performance required.
“Our first permissions implementation used JWTs, and it became obvious very quickly that scaling would be a problem,” said Gautam Pachnanda, Principal Engineer.
“We molded JWTs to our use cases at the app level, but we needed to drill down into project-based and environment-based permissions,” noted Lee Power, Product Manager. “The existing method just wasn’t fit for that purpose.”
Deciding on a Zanzibar-based authorization approach
The Matillion team knew they were building a complex permissions system that would need to grow with the platform. Before committing to building something new internally, they decided to explore existing offerings. One engineering manager already had his eye on a solution.
“I’d come across Zanzibar when I was working on a permissions system at my previous company,” said Rahat Khan, Engineering Manager. “In that role, I learned very quickly how difficult it was to implement a permissions system, and also how much Zanzibar brought to the table. I knew if there was a solution based on Zanzibar, it would be more viable than trying to build it ourselves.”
This proved to be a valuable perspective. The Matillion team spent time looking at various Zanzibar-based implementations. They knew they were on the right track when they found many businesses building products and commercial offerings based on it, including AuthZed.
“We knew Zanzibar was a well-proven model, but we also had a set of operational criteria we needed to fill,” noted Gautam. “There was the business need and the operational need. I wanted something that wasn’t coupled to an identity provider, had proven scalability that could sustain any growth our product experienced, that was flexible, without the constraints of JWTs, and that could run across multiple regions to a global scale. The solution had to be proven for the future. Zanzibar was a product that had been run at Google scale. We know that we won’t ever be at the scale of Google, but we also know that if it can run at that scale, we have reasonable assurance from an engineering perspective that this will stand the test of time for Matillion.”
Finding the right permissions system: SpiceDB Enterprise
It was key to get buy-in for the selected permissions system from internal security, engineering, and operations teams from the very start. Security and operations teams researched various authorization solutions, audited these systems, and decided what path they thought made sense.
“We gave the security and operational teams an independent opportunity to do their own audit of the various systems,” explained Gautam, “and asked them to please share their findings, which worked really well.”
Every team came to the same conclusion: SpiceDB Enterprise was the ideal foundation for Matillion. It had a proven record of scaling and could be self-hosted with enterprise support from AuthZed– the perfect combination of factors for Matillion’s forward-looking team.
With everyone aligned on SpiceDB Enterprise, the team began developing with their new fine-grained permissions capabilities. They designed environment-level permissions that allow certain roles to only alter development environments and not production ones. This is about as granular as you can get, and key for enterprise customers wanting greater control.
SpiceDB Enterprise was a simple choice for Matillion. Since permissions were such a critical part of their system, the team wanted to maintain a deep understanding of their technology stack and have the ability to run it themselves with the support of the experts at AuthZed who built and regularly deploy SpiceDB as an authorization service at scale.
“We knew we wanted to work with access control experts,” said Gautam. “The bit I enjoyed the most was the constant to and from around design decisions. Every design decision we made involved a chat with the AuthZed folks where we traded thoughts back and forth. That was really enjoyable, because we were able to validate whether the principles we were applying were suitable for SpiceDB.”
Permissions that power collaboration across data teams
Ultimately, adopting SpiceDB Enterprise empowered Matillion to successfully and efficiently build and launch an everyone-ready data platform. A flexible, secure, and performant permissions system was critical across the entire product.
“Our metrics of success are about adoption,” said Lee Power. “SpiceDB Enterprise is the underlying technology behind the features we present within the application. Nearly every one of our customers using the platform has asked for what we’ve built with project roles and environment permissions.”
Matillion’s adoption of SpiceDB Enterprise allowed their teams to easily ship a more robust product altogether. With SpiceDB in place, Matillion is ready to take on the future of data productivity!
