In today’s data-driven world, traditional monolithic data architectures are struggling to keep up with the demands of modern businesses.
They still serve their purpose and aren’t outdated. However, the steady progression towards cloud-based architectures offers significant improvements in volume, variety, velocity, and processing capacity. As a result, organizations in these environments are increasingly seeking more scalable, flexible, and agile approaches to manage their data ecosystems.
That’s where Data Mesh comes in.
Data mesh is a methodology that moves away from the data centralisation paradigm that has dominated the data modelling ecosystem for many years. Instead, it opts for decentralised data management, which empowers domain teams to own and operate their own data stores as a product.
In this blog, we’ll explore how you could implement Data Mesh methodology within Calibo and how doing so can enhance your data ecosystem by building data repositories that are semantically and contextually fitted to your business.
What is Data Mesh?
Tech salespeople love talking about it more than Reddit loves talking about Jared Leto, but what actually is it?
Data Mesh is the new cool kid on the block in the world of data and analytics. It’s an architectural approach to data warehousing that promotes a self-serve data infrastructure by allowing each respective team to ‘own’ their own data store. Coined by Zhamak Dehghani, the core principles of data mesh include:
- Domain-oriented decentralized data ownership and architecture: Data is managed by cross-functional teams closest to the data source and domain knowledge.
- Data as a product: Data sets are treated as products with clear ownership, quality standards, and discoverability.
- Self-serve data infrastructure as a platform: Providing tools and infrastructure that enable domain teams to build and manage their data products autonomously.
- Federated computational governance: Ensuring global standards, security, and interoperability across data products through a federated governance model.
Calibo & Data Mesh – a perfect fit
Calibo, on the other hand, is a software development and orchestration platform that brings together all the best-in-breed tools and technologies to build your digital product seamlessly and under a ‘single pane of glass’.
This promotes team cohesion and helps facilitate proper governance. Think of it as the composer who brings together the cacophony of instruments to help you write the perfect symphony.
How do I combine Calibo and Data Mesh?
- Establish your guardrails on Calibo. As mentioned, Calibo provides the tools you need to promote proper governance and oversight over your data product lifecycle. Define tooling, practices, and policies once and reuse them throughout the lifecycle, and use Calibo’s role-based access to manage your team’s access. This is an important consideration of data mesh, ‘federated governance.’
- Create fusion teams. Create and onboard fusion teams from users brought into Calibo through your integrated IDP. Creating fusion teams is a key benefit to working with Calibo with the focus on enabling a ‘single pane of glass’ view of your infrastructure.
- Fusion teams are a core tenant of data mesh as the data stores you require must be contextually useful and relevant, as well as technically viable. So,
we must bring business folk in to provide the context and help translate the business needs for the technical developers who will implement the data. This is where we satisfy the first tenant of Data Mesh, domain oriented decentralised data ownership.
- Create your data integrations. Build your data integrations, bring your best-in-breed data platforms into Calibo and provide your fusion teams with the required access to the respective integrations. Utilise Calibo’s teams and role access functionality to control access to integrations.
- Build your self-serve data infrastructure. Calibo provides the DFS (Data Fabric Studio) for just this task! Bring your integrations into the DFS and create drag and drop data nodes from the integrations that map to tables or use Calibo’s assistive functionality, such as the data catalog tool, to create collections of data from your existing data catalogs.
- Add transformations and quality checks from within Calibo ensuring the guardrails around your governance processes extend all the way down to the row level. With Calibos ‘single pane of glass’ approach, it’s quick and easy for other members of your organisation to request access to the data products you build.
- You can even build pipeline templates that others can import (given the correct permissions!) so they can incorporate them into their own offerings – enabling ‘self
-service data’.
- Orchestrate your pipelines. Calibo is built to enable products, from start to finish. Once you’ve built your pipeline you can use publish it to leverage Calibo’s orchestration capabilities, scheduling automated runs and quality checks. Publish your data to a data warehouse or use Calibo to host API services to productise the data access, making it available for consumers (a.k.a. data as a product).
- Monitoring and improving. Calibo’s pipeline monitoring capabilities ensure that you have assurance of quality for the entire data lifecycle. Inside the DFS, you can build in the data tests that will give you the peace of mind that the data downstream is of the utmost integrity. Additionally, set notifications up with published data pipelines that will notify somebody when a run is unsuccessful.
So that’s it – a high-level overview of how you can implement data-mesh within Calibo. This results in a scalable, flexible, and self-serve data infrastructure that empowers domain teams to provide a source of truth that aligns with the business.
Conclusion
This has been a high-level overview but if you’d like a step-by-step worked example then please do get in touch.
To recap – creating a ‘Data Mesh’ involves decentralizing data ownership, treating data as a product, building self-serve infrastructure, implementing robust governance, enabling easy data access, and fostering a collaborative culture. The thing that people often misunderstand about data mesh is that it goes far beyond a simple data model and is, in fact, a complete architecture that democratises data for your organisation and emphasises business context over technical implementation.
Heck, go wild – use a Kimball model to model the data for your business domain!
In any case – hopefully by following these steps, you have a better understanding of the steps required to lay the foundation for a scalable, flexible, and efficient data management approach that aligns with the principles of Data Mesh.
Are you tired of trying to keep the overview in a messy data landscape, or waiting to get access to the infrastructure and data sources you need to start developing? Learn more about Calibo Data Fabric here.
