Written by Emma Bromet & Rich Glew
Held over four days from June 10 to 13 in San Francisco, the Databricks Data + AI Summit 2024 brought together more than 16,000 data professionals, academics, and business leaders for keynotes, breakout sessions, lightning talks, and more. This year’s summit highlighted Databricks’ ongoing commitment to open-source initiatives and showcased several significant product announcements.
Themes & takeaways
Open source commitment
Databricks’ dedication to open-source solutions was reaffirmed, with significant announcements such as the acquisition of Tabular and the open-sourcing of the Unity catalogue. These moves highlight Databricks’ ongoing mission to make data management more accessible and collaborative.
Every company is a Data & AI Company
One of the summit’s central themes was the assertion that every organisation, regardless of industry, is fundamentally a data and AI company. Databricks stressed the importance of data literacy and accessibility, unveiling products designed to democratise data usage across all organisational levels.
Natural Language and No-Code Tooling
In line with Databricks’ ambition to “democratise data”, new natural language and no-code tools were introduced, with the aim of empowering all employees, not just data engineers, to engage with data and foster a more inclusive data-driven culture.
A Focus on Generative AI
As expected, Generative AI was a hot topic, with Databricks emphasising the need for large language models (LLMs) to be built on high-quality data, rather than using the biggest, most powerful models available. They highlighted the significance of compound systems, which encompass model training, evaluation, AI guardrails, and governance controls, ensuring robust and ethical AI deployment.
A Unity Catalog Approach
Databricks has positioned its Unity Catalog (UC) as a comprehensive solution for managing a safe and trusted data platform. The UC’s interoperability with other data systems aims to prevent vendor lock-in, allowing companies to bring their compute to the data – any compute – and focus on competitive execution.
“For example, in a world where file formats and governance are a solved problem, it means you can bring Starburst, Snowflake, BigQuery or whatever compute of your choice to your data, without having to move it. This is a massive and important development as, for most companies, 90% of their overall budget is spent moving and managing data and 10% on actual use cases. It’s why our platform builds are long and our analytics projects tend to be short.”
Emma Bromet
Partner, Data at Mantel Group
What does this mean for you?
Simplified Data Governance
With Databricks’ enhanced governance tools, managing data across diverse sources becomes more straightforward. This advancement reduces the time spent resolving fragmented data issues, allowing businesses to focus on securely and efficiently leveraging data for strategic outcomes.
Optimised FinOps and Compute Spend
The upcoming introduction of a 100% serverless architecture is poised to streamline networking and reduce costs, addressing a key concern for Chief Data Offers (CDOs) and Chief Financial Officers (CFOs). Despite these improvements, the need for comprehensive visibility and reporting on costs remains crucial, even in a 100% serverless world.
Accessible Generative AI
The summit highlighted the vast potential of Generative AI, particularly in customer service and marketing. Tools such as Databricks Agent can create personalised marketing campaigns with minimal effort, incorporating ‘human-in-the-loop” processes and inbuilt integrations, and further improving customer engagement and operational efficiency.
Enhanced Security and Governance
Security and Governance continue to be focus areas, with Databricks’ integrated tools supporting these needs, from design through to deployment. This framework ensures that Generative AI projects can progress beyond the proof-of-concept stage, and deliver real-world value to customers.
What lies ahead?
Databricks’ Data + AI Summit 2024 showcased a bold vision for the future of Data and AI. By addressing bottlenecks in data governance and AI deployment, Databricks is enabling customers to move quickly from initial implementation to realising tangible business benefits This will help organisations deliver real business value sooner, by taking time and effort spent on data foundations and moving to building value-add data and AI products instead. To hear more about our thoughts on what lies ahead for data and AI in 2025 and beyond, and how your business can use data and AI to drive growth and innovation, get in touch with our team of data experts today.
