There were a number of Microsoft Fabric announcements at Microsoft Build yesterday that I wanted to blog about.

Everything announced at Build can be found in the Microsoft Build 2024 Book of News.

Top announcement: Real-Time Intelligence

The new Real-Time Intelligence within Microsoft Fabric will provide an end-to-end software as a service (SaaS) solution that will empower customers to act on high volume, time-sensitive and highly granular data in a proactive and timely fashion to make faster and more-informed business decisions. Real-Time Intelligence, now in preview, will empower user roles such as everyday analysts with simple low-code/no-code experiences, as well as pro developers with code-rich user interfaces.

Features of Real-Time Intelligence will include:

• Real-Time hub, a single place to ingest, process and route events in Fabric as a central point for managing events from diverse sources across the organization. All events that flow through Real-Time hub will be easily transformed and routed to any Fabric data stores.
• Event streams that will provide out-of-the-box streaming connectors to cross cloud sources and content-based routing that helps remove the complexity of ingesting streaming data from external sources.
• Event house and real-time dashboards with improved data exploration to assist business users looking to gain insights from terabytes of streaming data without writing code.
• Data Activator that will integrate with the Real-Time hub, event streams, real-time dashboards and KQL query sets, to make it seamless to trigger on any patterns or changes in real-time data.
• AI-powered insights, now with an integrated Microsoft Copilot in Fabric experience for generating queries, in preview, and a one-click anomaly detection experience, allowing users to detect unknown conditions beyond human scale with high granularity in high-volume data, in private preview.
• Event-Driven Fabric will allow users to respond to system events that happen within Fabric and trigger Fabric actions, such as running data pipelines.

More info
Documentation

Other announcements

Updates to Fabric include:

• Fabric Workload Development Kit: When building an app, it must be flexible, customizable and efficient. Fabric Workload Development Kit will make this possible by enabling ISVs and developers to extend apps within Fabric, creating a unified user experience. This feature is now in preview. More info, documentation
• Fabric Data Sharing feature: Enables real-time data sharing across users and apps. The shortcut feature API allows seamless access to data stored in external sources to perform analytics without the traditional heavy integration tax. The new Automation feature now streamlines repetitive tasks resulting in less manual work, fewer errors and more time to focus on the growth of the business.These features are now generally available.
• GraphQL API and user data functions in Fabric: GraphQL API in Fabric is a savvy personal assistant for data. It’s a RESTful API that will let developers access data from multiple sources within Fabric, using a single query. User data functions will enhance data processing efficiency, enabling data-centric experiences and apps using Fabric data sources like lakehouses, data warehouses and mirrored databases using native code ability, custom logic and seamless integration. These features are now in preview.
• AI skills in Fabric: AI skills in Fabric is designed to weave generative AI into data specific work happening in Fabric. With this feature, analysts, creators, developers and even those with minimal technical expertise will be empowered to build intuitive AI experiences with data to unlock insights. Users will be able to ask questions and receive insights as if they were asking an expert colleague while honoring user security permissions. This feature is now in preview.
• Copilot in Fabric: Microsoft is infusing Fabric with Microsoft Azure OpenAI Service at every layer to help customers unlock the full potential of their data to find insights. Customers can use conversational language to create dataflows and data pipelines, generate code and entire functions, build machine learning models or visualize results. Copilot in Fabric is generally available in Power BI and available in preview in the other Fabric workloads. More info
• Snowflake Apache Iceberg shortcuts in Fabric: Apache Iceberg is an open-source native table format. With Iceberg shortcuts in OneLake, users will be able to unify data across domains, clouds and accounts by creating a single virtual data lake for the entire enterprise. Through Iceberg shortcuts, now in preview, Microsoft Fabric customers will be able to connect Iceberg tables in Snowflake to Fabric quickly, easily and without compromising performance. This is possible because of the Apache XTable project that both Microsoft and Snowflake are committed to (see Snowflake and Microsoft announce expansion of their partnership).  Previously shortcuts only supported Delta format.
• Direct Snowflake integration: In Snowflake, when creating a Snowflake database, you can specify to use iceberg tables (instead of Snowflake propriety format) and also specify that you will use Microsoft Fabric as the external storage provider, resulting in data that will be stored in Iceberg format in OneLake. So in OneLake you will see a Snowflake database that has been automatically created. Within Fabric, you can then create shortcuts inside this Snowflake database to other OneLake items (like a lakehouse or warehouse) that will be expressed in Iceberg format. This means if I go into the Snowflake environment and look inside this newly created database, I will see those OneLake items that have shortcuts to them. The result: I can query this Snowflake database in Fabric and updates made to it via Snowflake will be seen immediately in Fabric. Also, when using the database within Snowflake, I can combine the data with those Fabric items I have created shortcuts to. This will be preview later this year.
• Data workflows in Microsoft Fabric: Powered by Apache Airflow runtime, this will help you author, schedule, and monitor workflows or data pipelines using Python. More info
• Azure Databricks Unity Catalog integration with Fabric: Coming soon, you will be able to access Azure Databricks Unity Catalog tables directly in Fabric, making it even easier to unify Azure Databricks with Fabric. From the Fabric portal, you can create and configure a new Azure Databricks Unity Catalog item in Fabric with just a few clicks. You can add a full catalog, a schema, or even individual tables to link and the management of this Azure Databricks item in OneLake—a shortcut connected to Unity Catalog—is automatically taken care of for you. This data acts like any other data in OneLake—you can write SQL queries or use it with any other workloads in Fabric including Power BI through Direct Lake mode. When the data is modified or tables are added, removed, or renamed in Azure Databricks, the data in Fabric will remain always in sync. This new integration makes it simple to unify Azure Databricks data in Fabric and seamlessly use it across every Fabric workload.
• Federate OneLake as a Remote Catalog in Azure Databricks: Also coming soon, Fabric users will be able to access Fabric data items like lakehouses as a catalog in Azure Databricks. While the data remains in OneLake, you can access and view data lineage and other metadata in Azure Databricks and leverage the full power of Unity Catalog. This includes extending Unity Catalog’s unified governance over data and AI into Azure Databricks Mosaic AI. In total, you will be able to combine this data with other native and federated data in Azure Databricks, perform analysis assisted by generative AI, and publish the aggregated data back to Power BI—making this integration complete across the entire data and AI lifecycle.

More info

Also check out the Microsoft Fabric May 2024 Update