There is a revolution in organisational design development brewing in the data industry. Borrowing from the same mantra of ‘decentralisation’ that has sparked innovation in blockchain technology and Web3, “Data Mesh” is the new kid on the block that has caused a stir amongst data evangelists and business thinkers alike.
The thinking behind Data Mesh is very appealing. By distributing capabilities into clearly defined data products and leveraging best practices in technology delivery, teams are empowered to take ownership over their data and realise the dream of self-service analytics. This is not only a business imperative for technology leaders looking to the future of data management, but also for all those aspiring tech gurus within an organisation that want a slice of the action.
By getting rid of an organisation’s centralised data team, individuals will be inspired to take a greater creative role and will lead to product decisions that are more data-driven and agile in nature.
This is power to the (data) people. However, it’s easy to get swept up in the new and shiny. Data Mesh is no light undertaking and the process will involve a complete re-imagination of an organisation’s practices, causing disruption and even possible resistance from within! As a good revolutionary knows, any successful challenge to the status-quo requires significant buy-in and moreover, mass organisation. The merits of Data Mesh must be evaluated against the backdrop of an organisation’s capabilities, time and resources, before any commitments are made.
Okay, so what are the potential pitfalls of Data Mesh?
For the most part, the technology to make Data Mesh a success already exists. Some may argue that a complete solution has not yet fully presented itself in the market, however this is to distract at the core of what Data Mesh is and where the problems may be. It is, first and foremost, a ‘socio-technical’ paradigm concerned with organisational change, a differentiating feature from other management strategies. This is a good time to remind ourselves of Data Mesh’s 4 main principles: domain-driven data ownership, data viewed as a product, self-service data platforms and a federated governance function.
Of course, the adoption of cloud storage, data lakes, management and analytical tools go hand in hand with a sound implementation and whilst this is important to get right, it should be noted that Data Mesh’s driving philosophy is that of social evolution.
For this reason, the main issues that often arise from the roll-out of Data Mesh are either cultural, governance related, or managerial in nature.
How to implement data mesh
Create a team culture that inspires
It’s tempting to prematurely brand your organisation as more tech-oriented – buying tickets for the data train is the easy part, convincing everyone to jump on board is a lot harder. There are many audiences to consider, from sales, marketing, procurement to finance and all with varying levels of data literacy and competency. Each will need to play a part, though some may struggle to adapt to the challenges. The mere suggestion of “restructure” could send some running for the hills.
So, how ready is an organisation? This may be a good litmus test. How many would empathise and maybe chuckle with the meme below?
The answers may be revealing. If individuals are not aware of the pain-points that current data architectures may have, the virtues of Data Mesh will be lost on them and it’s essential that they too form part of the solution. Culture eats strategy for breakfast and if fully embraced into a concise, long-term vision for the firm, the weave of the Mesh can strengthen bonds between departments and create a more data-conscientious workforce.
Gain sponsorship for Data Mesh from up top
This line of thinking also extends to management who require the highest level of engagement, though ironically this group may be the trickiest player to convince. In order to create a true prototype of Data Mesh, the organisation and all its machinations will need to be broken down into a fine sand to then later be redrawn. This mindset shift ultimately has to be the responsibility of senior leadership.
It is recommended that Data Mesh should be championed across the business, even though it may be tempting to trial the model in an isolated part of the organisation. Given the level of investment required, it does make sense to involve as many parts of the business as early as possible, where the full benefits of the Mesh at scale can be seen. This is not to say that best practice hybrid models will not emerge in the future.
Governing with the new sheriffs in town
Organisational politics are never a pleasant subject to broach, however they become inevitable when roles and governance functions are redistributed. How many data team leads would get excited about the prospects of now performing a role with potentially more oversight and less control over the data function? It’s an unavoidable truth that Data Mesh will relinquish many of the responsibilities that a centralised data team would have once enjoyed. Instead, new independent domains can take the reins and it’s easy to see how heads can knock in this power dynamic. Navigating this will be crucial.
The question of accountability also becomes an issue. For instance, if there is a bug that spans multiple products, or a conflict that arises between domains, who will handle the administration of these issues? It’s also likely that team budgets would have seen a shake up in the transition to Data Mesh and will inevitably dirty the waters as to who bears responsibility.
Investing in good people in data engineering, governance and analysis to get the job done
Self-service frameworks are a natural progression towards advanced analytics, however it must be delivered by teams that are capable of the task. It’s been observed that organisations may have been too quick in embracing tools, without much thought into its people. In the worst of cases, it has created tech bloat and mistrust in data products that have been delivered by teams that do not have the technical-savviness to engender confidence between domains. That new barista endorsed coffee machine sitting in the office kitchen may look great, but it’s not going to create a good cup of coffee, unless it’s flashy buttons are studied and pressed in the right order.
Organisations, irrespective of size or industry, may be at different points on the data maturity curve. What more, differences may exist between departments themselves and this is to be expected. There may be some areas of the business that would have never touched a tech stack in their lives! The question of resourcing becomes important. Each domain must be able to operate independently from each other to manage all of their data needs, which means a minimum investment in data engineering, governance and analysis capabilities.
Data Mesh could be the solution, but is your organisation ready?
Big data is transforming the technology landscape as we see it. Practices in organisational design must in turn keep up and any business that is serious in using data to its full benefit, should consider Data Mesh as a candidate. However, it must be acknowledged that there are serious challenges in this journey and any advocacy for its adoption must be contextualised, given an organisation’s cultural capital, long-term strategic goals and desire for change. This can avoid the Mesh turning into a mess.
Getting help
This is a bit of a shameless plug but if you think you have a challenge with your Data Mesh model or are unsure of where to start, we at Mantel Group can help you. Please get in touch if you want to know more. Mantel Group has successfully helped multiple organisations navigate the technical and organisational complexities involved with Data Mesh architectures.
Written by
Stefan Fernandes – Data Consultant
Mantas Zegeris-Kaleda – Senior Data Consultant
