4 years in the past, Databricks noticed large complexity within the knowledge panorama: separate catalogs for every platform, siloed governance instruments throughout clouds, and no unified approach to safe AI belongings. We pioneered Unified Governance by launching Unity Catalog, an open, versatile catalog layer to handle entry, lineage, auditing, and discovery throughout all knowledge and AI belongings.
In the present day, Unity Catalog has change into the muse of the Databricks Knowledge Intelligence Platform and the business’s solely unified governance answer for knowledge and AI throughout codecs, clouds, and engines. From open knowledge sharing to fine-grained safety and data governance, Unity Catalog helps organizations convey context, management, and confidence to their knowledge property.
At this yr’s Knowledge + AI Summit, we’re asserting main improvements throughout Unity Catalog, delivering the most effective catalog for Apache Iceberg™, new enterprise person experiences, and clever governance to guard delicate knowledge and guarantee trusted knowledge high quality at scale.
Right here’s what’s new.
The Greatest Catalog for Apache Iceberg™
Organizations adopting a lakehouse are sometimes pressured to decide on between Delta Lake and Apache Iceberg™. That selection creates synthetic silos: proscribing entry to the information and AI instruments that groups can use, fragmenting governance, and locking metadata into format-specific catalogs.
Unity Catalog eliminates the necessity to decide on. Constructed on open requirements, Unity Catalog is the one unified catalog that works seamlessly throughout codecs, engines, and clouds, making it the muse of the open lakehouse. Over the previous yr, following the acquisition of Tabular, we’ve invested deeply in Apache Iceberg to increase this imaginative and prescient. We’re excited to announce:
- Full help for the Iceberg REST Catalog API, permitting exterior engines to learn (Usually Out there) and write (Public Preview) to Unity Catalog–managed Iceberg tables. This can be a main differentiator out there, eliminating format lock-in and enabling full interoperability unmatched by another answer.
- Iceberg managed tables at the moment are in Public Preview, delivering best-in-class worth and efficiency, liquid clustering, predictive optimization, and full integration with Databricks and throughout exterior engines, together with Trino, Snowflake, and Amazon EMR.
- Iceberg catalog federation is in Public Preview, enabling you to manipulate and question Iceberg tables managed in AWS Glue, Hive Metastore, and Snowflake Horizon with out copying knowledge.
- Delta Sharing for Iceberg is now in Non-public Preview, permitting you to share Unity Catalog tables and Delta tables with any recipient utilizing Delta Sharing and devour them in any shopper that helps the Iceberg REST Catalog API.
Collectively, these capabilities break down format silos and set Unity Catalog aside as the one catalog that delivers actually open, unified governance and interoperability. Take a look at our weblog on Iceberg help to study extra about these bulletins.
Increasing Unity Catalog to enterprise customers
Knowledge platforms shouldn’t cease on the technical person. Enterprise customers want a transparent, constant approach to discover, belief, and work with knowledge. Unity Catalog now presents a unified basis for enterprise context to bridge the hole between knowledge and enterprise groups.
Unity Catalog Metrics: One semantic layer for all knowledge and AI workloads
Inconsistent metric definitions throughout instruments and groups have lengthy brought about confusion, misalignment, and an absence of belief in knowledge. Unity Catalog Metrics, now in Public Preview on AWS, Azure, and GCP and Usually Out there later this summer season, solves this by making enterprise metrics first-class belongings within the lakehouse. Not like metrics outlined solely within the BI layer, which restrict reuse and integration, defining metrics on the knowledge layer makes enterprise semantics reusable throughout all workloads, like dashboards, AI fashions, and knowledge engineering jobs. Unity Catalog Metrics are additionally totally addressable by way of SQL to make sure that everybody within the group can have the identical view of metrics, regardless of what software they select.
- Outline as soon as, use in all places: Create metrics as soon as in Unity Catalog and use them throughout AI/BI Dashboards, Genie, Notebooks, SQL, and Lakeflow jobs. Upcoming integrations will prolong help to BI instruments like Tableau, Hex, Sigma, ThoughtSpot, Omni and observability instruments like Anomalo and Monte Carlo.
- Ruled and auditable by default: Licensed metrics include auditing and lineage out of the field, enabling trusted, compliant insights throughout groups.
“Unity Catalog Metrics offers us a central place to outline enterprise KPIs and standardize semantics throughout groups, guaranteeing everybody works from the identical trusted definitions throughout dashboards, SQL, and AI purposes.”
— Richard Masters, Vice President, Knowledge & AI, Virgin Atlantic
“Unity Catalog Metrics represents an thrilling alternative for Tableau prospects to leverage the worth of centralized governance with Databricks Unity Catalog. By means of our deep integration and increasing roadmap with Databricks, we’re thrilled to assist take away the friction for our prospects in leveraging Databricks to outline their core enterprise metrics.”
— Nicolas Brisoux, Sr. Director Product Administration, Tableau
New curated discovery experiences with clever insights
To totally empower enterprise customers, you should make trusted knowledge straightforward to search out, perceive, and use. Unity Catalog is extending its business-aware governance with a brand new Uncover expertise, now in Non-public Preview, a curated inside market of licensed knowledge merchandise organized by enterprise domains like Gross sales, Advertising and marketing, or Finance.
AI-powered suggestions and knowledge steward curation assist floor the highest-value belongings, corresponding to metrics, dashboards, tables, AI brokers, and Genie areas which are enriched with documentation, possession, and utilization insights. New clever alerts spotlight knowledge high quality, utilization patterns, relationships, and certification standing, serving to customers rapidly assess belief and relevance. Plus, with Databricks Assistant inbuilt, customers can ask pure language questions and get clear, context-aware solutions based mostly on ruled metrics.
We’re additionally introducing new clever capabilities throughout Databricks to make knowledge discovery simpler and extra intuitive, wherever customers work within the platform. Powered by Unity Catalog, these options assist groups discover trusted knowledge quicker and perceive its context at a look.
- Domains (Coming quickly): Set up knowledge by enterprise space to align discovery with the group’s operations.
- Certifications and Deprecation Tags (Beta): Sign knowledge belief and enterprise relevance throughout datasets, metrics, and dashboards. Tagged belongings prominently show their standing in authoring surfaces just like the SQL editor, preserving knowledge high quality alerts seen all through the person workflow. Certifications and deprecation tags can be found as part of Tag Insurance policies Beta.
- Request for Entry (Public Preview): To streamline supply, customers can immediately request knowledge entry on to the asset.
Extra superior governance capabilities now obtainable
Excessive-leverage governance with scalable, attribute-driven controls
Scaling knowledge governance turns into more and more difficult as organizations develop, with extra customers, groups, and knowledge belongings to handle. Static insurance policies and handbook controls can’t sustain, resulting in governance gaps, safety dangers, and operational bottlenecks.
To deal with these challenges, Unity Catalog now gives clever automation and versatile, scalable controls to categorise delicate knowledge, implement coverage persistently, and speed up safe knowledge entry throughout the lakehouse.
-
Attribute-based entry management (ABAC): Outline versatile entry insurance policies utilizing tags that may be utilized on the catalog, schema, or desk stage. ABAC is offered in Beta for row and column-level safety on AWS, Azure, and GCP.
-
Tag insurance policies: Tag insurance policies implement a governance layer for the way tags are created, assigned, and used throughout Databricks. These account-level insurance policies guarantee tags stay constant and trusted, supporting every thing from knowledge classification to value attribution. Tag insurance policies can be found in Beta on AWS, Azure, and GCP.
-
Knowledge classification: Intelligently detect and tag delicate knowledge throughout Unity Catalog. New knowledge is scanned inside 24 hours to routinely detect new PII, minimizing handbook effort and permitting groups to remain on prime of information entry. When used with ABAC, Knowledge classification routinely protects delicate knowledge based mostly in your entry management insurance policies. Knowledge classification is offered in Beta on AWS, Azure, and GCP.
“Implementing column masking throughout greater than 5,000 tables was once an unlimited handbook effort. With ABAC, we’re in a position to apply constant insurance policies dynamically, drastically enhancing each velocity and governance.”
— Ramesh Balasubramanyan, Databricks Admin, SAIF
“Databricks Knowledge Classification has been a game-changer in our knowledge privateness and safety technique. Paired with ABAC, it permits us to routinely safe delicate knowledge with out proscribing the information that our analysts want. The largest profit has been velocity, with automated classification and masking considerably lowering handbook overhead, releasing up our resourcing and saving our crew numerous hours every week.”
— Mary Tesfay, Knowledge & Analytics Lead, Corp IT, Navitas
Automated knowledge high quality monitoring at scale
Unity Catalog now intelligently detects and helps resolve knowledge high quality points throughout all of your tables with knowledge high quality monitoring, obtainable in beta on AWS, Azure, and GCP. Knowledge high quality monitoring checks freshness—how not too long ago knowledge has been up to date—and completeness—whether or not knowledge volumes are as anticipated—utilizing knowledge intelligence throughout complete schemas. Customers are in a position to perceive the well being of information at a look with well being indicators, whereas knowledge homeowners can perceive the precedence of points based mostly on downstream lineage, uncover the basis trigger, and set alerts utilizing built-in logging and dashboards.
Get began with Unity Catalog, the muse of Knowledge Intelligence
Unity Catalog continues evolving because the business’s solely unified governance layer, the muse for safe, clever, and business-aware knowledge platforms. Whether or not you’re constructing AI brokers, delivering BI dashboards, or sharing knowledge throughout organizations, Unity Catalog connects all of it by a single, open catalog.
To get began, observe the Unity Catalog guides for AWS, Azure, and GCP.
Watch the Knowledge + AI Summit 2025 keynote from Matei Zaharia, Co-founder and Chief Know-how Officer at Databricks, to study extra about these latest bulletins.
Register for Knowledge + AI Summit and discover the knowledge and AI governance observe
