Data mesh

From Wikipedia the free encyclopedia

Data mesh is a sociotechnical approach to building a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] Data mesh mainly concerns itself with the data itself, taking the data lake and the pipelines as a secondary concern. [3] The main proposition is scaling analytical data by domain-oriented decentralization.[4] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[5] This enables a decrease in data disorder or the existence of isolated data silos, due to the presence of a centralized system that ensures the consistent sharing of fundamental principles across various nodes within the data mesh and allows for the sharing of data across different areas.[6]

History[edit]

The term data mesh was first defined by Zhamak Dehghani in 2019[7] while she was working as a principal consultant at the technology company Thoughtworks.[8][9] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[10][11] Data meshes have been implemented by companies such as Zalando,[12] Netflix,[13] Intuit,[14] VistaPrint, PayPal[15] and others.

In 2022, Dehghani left Thoughtworks to found Nextdata Technologies to focus on decentralized data.[16]

Principles[edit]

Data mesh is based on four core principles:[17]

In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[19] In other words, the data should be treated as a product that is ready to use and reliable.[20]

Data mesh in practice[edit]

After its introduction in 2019[7] multiple companies started to implement a data mesh[12][14][15] and share their experiences. Challenges (C) and best practices (BP) for practitioners, include:

C1. Federated data governance
Companies report difficulties to adopt a federated governance structure for activities and processes that were previously centrally owned and enforced. This is especially true for security, privacy, and regulatory topics.[21][22][23]
C2. Responsibility shift
In data mesh individuals within domains are end-to-end responsible for data products. This new responsibility can be challenging, because it is rarely compensated and usually benefits other domains.[21][22]
C3. Comprehension
Research has shown a severe lack of comprehension for the data mesh paradigm among employees of companies implementing a data mesh.[21]
BP1. Cross-domain unit
Addressing C1, organizations should introduce a cross-domain steering unit responsible for strategic planning, use case prioritization, and the enforcement of specific governance rules—especially concerning security, regulatory, and privacy-related topics. Nevertheless, a cross-domain steering unit can only complement and support the federated governance structure and may grow obsolete with the increasing maturity of the data mesh.[21][24]
BP2. Track and observe
Addressing C2., organizations should observe and score data product quality as tracking and ranking key data products can encourage high-quality offerings, motivate domain owners, and support budget negotiations.[21]
BP3. Conscious adoption
Organizations should thoroughly assess and evaluate their existing data systems, consider organizational factors, and weigh the potential benefits before implementing a data mesh. When introducing data mesh, it is advised to carefully and consciously introduce data mesh terminology to ensure a clear understanding of the concept (C3).[21]

Community[edit]

Scott Hirleman has started a data mesh community that contains over 7,500 people in their Slack channel.[25]

See also[edit]

References[edit]

  1. ^ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
  2. ^ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.{{cite book}}: CS1 maint: location missing publisher (link)
  3. ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2022-01-01). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. hdl:1822/78127. ISSN 1877-0509. S2CID 245864612.
  4. ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
  5. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  6. ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2022-01-01). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. hdl:1822/78127. ISSN 1877-0509.
  7. ^ a b "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
  8. ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
  9. ^ Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
  10. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  11. ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
  12. ^ a b Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
  13. ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
  14. ^ a b Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
  15. ^ a b "The next generation of Data Platforms is the Data Mesh". 2022-08-03. Retrieved 2023-02-08.
  16. ^ "Why We Started Nextdata". 2022-01-16. Retrieved 2023-02-08.
  17. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  18. ^ "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
  19. ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
  20. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  21. ^ a b c d e f Bode, Jan; Kühl, Niklas; Kreuzberger, Dominik; Hirschl, Sebastian; Holtmann, Carsten (2023-05-04). "Data Mesh: Motivational Factors, Challenges, and Best Practices". arXiv:2302.01713v2 [cs.AI].
  22. ^ a b Vestues, Kathrine; Hanssen, Geir Kjetil; Mikalsen, Marius; Buan, Thor Aleksander; Conboy, Kieran (2022). "Agile Data Management in NAV: A Case Study". Agile Processes in Software Engineering and Extreme Programming. Lecture Notes in Business Information Processing 445 LNBIP. Vol. 445. Springer. pp. 220–235. doi:10.1007/978-3-031-08169-9_14. ISBN 978-3-031-08168-2.
  23. ^ Joshi, Divya; Pratik, Sheetal; Rao, Madhu Podila (2021). "Datagovernanceindata mesh infrastructures: The Saxo bank case study". Proceedings of the International Conference on Electronic Business (ICEB). Vol. 21. pp. 599–604.
  24. ^ Whyte, Martin; Odenkirchen, Andreas; Bautz, Stephan; Heringer, Agnes; Krukow, Oliver (2022). "Data Mesh - Just another buzzword or the next generation data platform?". PwC study 2022: Changing data platforms.
  25. ^ "The Global Home for Data Mesh". The Global Home for Data Mesh. Retrieved 2022-04-24.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Data_mesh&oldid=1206413529"