One thing I love about working in tech is that the landscape is constantly changing. Like the weeping angels in Dr Who – every time you turn back and look – the tech landscape has moved slightly.
Unlike the weeping angels, however – this progress is for the betterment of all. (And slightly less murderous). If you have been keeping up with the zeitgeist in the data space (or you just love reading my blogs!), you’ll no doubt have heard the terms ‘Data Fabric’, or ‘Data Mesh’ being thrown around.
In this blog, we will decipher what these terms mean and whether they are something you should pay attention to or another fad.
Let’s kick off with the basics…
What is data mesh?
Data Mesh is a methodology that decentralizes data ownership and management, aligning it with business domains.
It treats data as a product, giving domain teams the responsibility and autonomy to manage their data pipelines, storage, and governance. To achieve a data mesh, there are some core tenants to follow:
Core principles of data mesh
- Decentralized data ownership: Domains own and manage their data, ensuring it aligns with their specific needs.
- Data as a product: Data is treated with the same importance as software products, making it reliable, discoverable, and usable.
- Self-service data infrastructure: Domain teams have the tools they need to manage their data independently.
- Federated governance: Standards and policies are established across domains to ensure data quality and compliance
, without centralizing control.
When should you think about applying Data Mesh?
- Domain-oriented organisations: if your organisation has highly federated or decentralised teams that would benefit from autonomy over their data
- Requirement for high scalability: if your organization’s data must scale with organization growth and complexity
What is data fabric?
Data Fabric, on the other hand, is an architectural approach to provide a unified layer of data management across various environments, such as on-premises systems, cloud platforms and edge computing.
It integrates and automates data workflows, ensuring consistent data management and accessibility.
Core principles of data fabric:
- Unified data management: provides a centralised view and management of data across hybrid and multi-cloud environments.
- Data integration: Integrates data from diverse sources seamlessly, making it accessible through a single interface.
- Automation and orchestration: Automates data processes for the efficient movement, transformation and governance of data.
- Self-service access: Empowers users with self-service capabilities, reducing reliance on IT for data access.
- Scalability and flexibility: Scales with data growth and adapts to different data types and workloads.
When to apply data fabric
- Large, diverse data ecosystem: Ideal for organisations with a wide variety of data sources and environments.
- Centralized data governance: perfect for companies needing robust governance and compliance across their data.
- Hybrid and multi-cloud environments: Best for environments spanning on-premises, cloud computing and edge computing.
Core differences in approach
While both approaches, data mesh and data fabric, aim to tackle the complexities of the modern data ecosystem, they do so through different paradigms.
Data Mesh decentralises data ownership, promoting a federated (domain) oriented approach with much more team autonomy over their respective data repository.
Which makes sense, right?
Let’s say we put it in terms of a mechanic’s workshop—a traditional steel, oil, and sweat mechanic likely doesn’t have the same knowledge of an electrical vehicle’s engine as a specific electrical vehicle engineer would. It’s a different domain.
On the other hand, data fabric centralises data management creating a unified layer that integrates data from various sources into a single, cohesive system.
Think of it like your Ali Baba, Amazon or eBay, where instead of spending your money supporting local highstreets, we impulse buy gizmos made halfway across the world from a single platform. This is analogous to seamlessly integrating your disparate data sources into one single unitary system.
When should I use…
Data mesh
Data Mesh is best applied in organisations that have distinct business domains and units that require autonomy and are capable of managing their own data independently.
It is particularly suited to environments where data needs to be scaled quickly with the organization’s growth and complexity. The value in this is that individual teams or domains are empowered with their data.
They can innovate and respond to changes swiftly and easily as they can apply changes themselves, making them much more agile.
Data fabric
Data fabric is particularly suited for organisations with a diverse array of data sources that need to be in a single unified system. It works well for environments that could span various data sources between on-prem and cloud, where robust governance and compliance are required across all data applications.
The value of this is that you can create a much more holistic view of your organisation’s data.
You can also ensure tighter adherence to governance as quality, accessibility and consistency are managed centrally which reduces the risk associated with handling data across diverse environments.
Hold up – plot twist
So, data mesh and data fabric are different approaches to architecting your organisation’s data. Although they occupy a similar space, they represent distinct approaches to data management.
Data mesh is a design methodology focused on decentralised domain ownership, while Data fabric is more of an architecture aimed at creating a unified data management layer.
Interestingly, it is possible to use them together, cherry-picking aspects of one to enhance and empower the other. For instance, the robust governance frameworks from a data fabric implementation can be applied to federated data mesh data repositories.
This ensures a unified source of truth while encouraging more coherent and consistent data management practices across domains. In addition, the metadata garnered through a data fabric architecture can enhance data discoverability to support data mesh-enabled organisations.
Conclusion
Today’s data ecosystem is huge and convoluted—a sprawling mass of technologies careening around the digital ether. Trundling legacy systems propping up archaic but time-tested systems are interspersed with lightning-fast behemoth cloud platforms—digital dams retaining the entire contents of the universe behind their data stores.
Seamlessly tapping into such a wide array of sources can be a daunting and challenging task but one that both data fabric and data mesh seek to address in their own distinct way.
Data Mesh emphasises decentalisation, empowering the autonomy of domain-oriented teams and promoting scalability in highly federated organisations. Using this approach offers federated businesses looking to grow fast a highly agile approach to data management with minimal friction.
Conversely, data fabric takes a contrary approach offering a centralised, unified approach to data management. It does this by weaving together your data sources, however disparate, and storing them centrally.
It is, therefore, best suited to organisations looking to apply highly robust data governance measures, holistic data views and consistency across their cloud ecosystem.
Finally, we saw that because these approaches are slightly different in methodology and approach, they actually aren’t mutually exclusive so it’s possible for organisations to reap the benefits of both.
In the end, the choice between data mesh and data fabric depends on your specific needs, structure and data management objectives. But hopefully, you now have a better understanding of the value that data fabric and data mesh can provide.
Learn more about how Calibo’s Data Fabric Studio can help you make smarter decisions in a complex data ecosystem here.
