133万现金不翼而飞 银行女职员盗取巨款还赌债
 | This article may be too technical for most readers to understand. (January 2023) |
Data mesh is a sociotechnical approach to building a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] Data mesh mainly concerns itself with the data itself, taking the data lake and the pipelines as a secondary concern. [3] The main proposition is scaling analytical data by domain-oriented decentralization.[4] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[5] This enables a decrease in data disorder or the existence of isolated data silos, due to the presence of a centralized system that ensures the consistent sharing of fundamental principles across various nodes within the data mesh and allows for the sharing of data across different areas.[6]
History
[edit]The term data mesh was first defined by Zhamak Dehghani in 2019[7] while she was working as a principal consultant at the technology company Thoughtworks.[8][9] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[10][11] Data meshes have been implemented by companies such as Zalando,[12] Netflix,[13] Intuit,[14] VistaPrint, PayPal[15] and others.
In 2022, Dehghani left Thoughtworks to found Nextdata Technologies to focus on decentralized data.[16]
Principles
[edit]Data mesh is based on four core principles:[17]
- domain ownership;
- data as a product;
- self-serve data platform;
- federated computational governance.
In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[18] In other words, the data should be treated as a product that is ready to use and reliable.[19][20]
In practice
[edit]After its introduction in 2019[7] multiple companies started to implement a data mesh[12][14][15] and share their experiences. Challenges (C) and best practices (BP) for practitioners, include:
- C1. Federated data governance
- Companies report difficulties to adopt a federated governance structure for activities and processes that were previously centrally owned and enforced. This is especially true for security, privacy, and regulatory topics.[21][22][23]
- C2. Responsibility shift
- In data mesh individuals within domains are end-to-end responsible for data products. This new responsibility can be challenging, because it is rarely compensated and usually benefits other domains.[21][22]
- C3. Comprehension
- Research has shown a severe lack of comprehension for the data mesh paradigm among employees of companies implementing a data mesh.[21]
- BP1. Cross-domain unit
- Addressing C1, organizations should introduce a cross-domain steering unit responsible for strategic planning, use case prioritization, and the enforcement of specific governance rules—especially concerning security, regulatory, and privacy-related topics. Nevertheless, a cross-domain steering unit can only complement and support the federated governance structure and may grow obsolete with the increasing maturity of the data mesh.[21][24]
- BP2. Track and observe
- Addressing C2., organizations should observe and score data product quality as tracking and ranking key data products can encourage high-quality offerings, motivate domain owners, and support budget negotiations.[21]
- BP3. Conscious adoption
- Organizations should thoroughly assess and evaluate their existing data systems, consider organizational factors, and weigh the potential benefits before implementing a data mesh. When introducing data mesh, it is advised to carefully and consciously introduce data mesh terminology to ensure a clear understanding of the concept (C3).[21]
Community
[edit]Scott Hirleman has started a data mesh community that contains over 7,500 people in their Slack channel.[25]
See also
[edit]- Data product
- Data management
- Data platform
- Data vault modeling, method of data modeling with storage of data from various operational systems and tracing of data origin, facilitating auditing, loading speeds and resilience
- Data warehouse, a well established type of database system for organizing data in a thematic way
- ETL and ELT
References
[edit]- ^ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
- ^ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2025-08-07). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. hdl:1822/78127. ISSN 1877-0509. S2CID 245864612.
- ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2025-08-07.
- ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2025-08-07). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. hdl:1822/78127. ISSN 1877-0509.
- ^ a b "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
- ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2025-08-07.
- ^ Andy Mott (2025-08-07). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2025-08-07.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2025-08-07. Retrieved 2025-08-07.
- ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2025-08-07.
- ^ a b Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
- ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2025-08-07
- ^ a b Baker, Tristan (2025-08-07). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2025-08-07.
- ^ a b "The next generation of Data Platforms is the Data Mesh". 2025-08-07. Retrieved 2025-08-07.
- ^ "Why We Started Nextdata". 2025-08-07. Retrieved 2025-08-07.
- ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.
{{cite book}}: CS1 maint: location missing publisher (link) - ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2025-08-07. Retrieved 2025-08-07.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2025-08-07. Retrieved 2025-08-07.
- ^ "Looking Glass | 2024". Thoughtworks. Retrieved 2025-08-07.
- ^ a b c d e f Bode, Jan; Kühl, Niklas; Kreuzberger, Dominik; Hirschl, Sebastian; Holtmann, Carsten (2025-08-07). "Data Mesh: Motivational Factors, Challenges, and Best Practices". arXiv:2302.01713v2 [cs.AI].
- ^ a b Vestues, Kathrine; Hanssen, Geir Kjetil; Mikalsen, Marius; Buan, Thor Aleksander; Conboy, Kieran (2022). "Agile Data Management in NAV: A Case Study". Agile Processes in Software Engineering and Extreme Programming. Lecture Notes in Business Information Processing 445 LNBIP. Vol. 445. Springer. pp. 220–235. doi:10.1007/978-3-031-08169-9_14. ISBN 978-3-031-08168-2.
- ^ Joshi, Divya; Pratik, Sheetal; Rao, Madhu Podila (2021). "Data Governance in Data Mesh Infrastructures: The Saxo Bank Case Study". Proceedings of the International Conference on Electronic Business (ICEB). Vol. 21. pp. 599–604.
- ^ Whyte, Martin; Odenkirchen, Andreas; Bautz, Stephan; Heringer, Agnes; Krukow, Oliver (2022). "Data Mesh - Just another buzzword or the next generation data platform?". PwC study 2022: Changing data platforms.
- ^ "The Global Home for Data Mesh". The Global Home for Data Mesh. Retrieved 2025-08-07.