Data virtualization refers to a general technology approach of abstracting data away from physical data sources, including data warehouses, data lakes, application data, without having to copy or move it. Data virtualization solutions are generally grouped in with other data integration technologies including ETL and ELT approaches.
AtScale is sometimes grouped in with traditional data virtualization vendors (for some examples, see this Gigaom Radar report on Data Virtualization) based on the approach we use to connect a semantic layer to raw data sources. While data virtualization concepts developed somewhat in parallel to semantic layers, it is useful to examine the origins and primary use cases for pure-play data virtualization solutions. That makes it easier to understand how a semantic layer uniquely supports modern BI and analytics programs built on cloud data platforms.
Evolution of Data Virtualization
The term data virtualization originated in the late 2000s initially as a broader set of capabilities including query federation, a concept that had emerged in the 1980s. The basic approach was to construct a virtual, logical data warehouse providing a single interface to retrieve data from multiple databases without any “physical” data integration. Queries could be generated at the virtual level and “passed” through to the underlying databases, with results returned to the user or client making the request.
The approach has been used for analytics use cases, reporting, data mining, and application design. It sometimes is positioned as an alternative to building a data warehouse – most notably when positioned with a modern data lake infrastructure as a “Lake House.”
Along with adjacent ETL/ELT technologies, data virtualization has become a foundational technology in modern data infrastructure. Data virtualization has the advantage of not being subject to the laws of “data gravity,” since no physical data (except query results) are moved. The concept has become even more attractive with the rise of big data analytics and cloud data platforms, where it can be difficult (read: complicated, expensive, and slow) to physically move data for all data consumption use cases. The entire data integration category, including data virtualization, is an increasingly relevant component of the modern data fabric. When it comes to BI and analytics use cases however, traditional data virtualization approaches are limited in their ability to support performant, multi-dimensional analysis.
Supporting Multiple BI Tools with Data Virtualization
Most organizations are running three or four BI tools at once, and nobody planned it that way. Power BI took hold in finance, Tableau got embedded in the data science team, and now someone wants to add Looker. Each one works fine on its own. Keeping them consistent with each other is where things get messy.
Data virtualization provides a shared access layer so tools can all pull from the same underlying data. Teams stop rebuilding the same business logic for every new platform, and metrics stop drifting apart depending on which tool generated the report.
It also means adding a new BI tool doesn’t require a migration project. The data stays where it is, and the new tool plugs into the same layer everyone else is already using.
The Co-Evolution of the Analytics Semantic Layer
The term semantic layer emerged in early 1990s with the original Business Objects patent describing the concept as a way of providing “a new data representation and a query technique, which allows information system end users to access (query) relational databases without knowing the relational structure or the structure query language (SQL).”
This approach established a whole school of thought around making dimensional analysis accessible to business analysts with limited understanding of database structure. It is easy to see the parallels to the origins of data virtualization – both focused on providing an alternative user interface to raw data in order to facilitate consumption and hide the underlying complexity of the raw data.
Traditional OLAP solutions ingest raw data and store dimensional aggregations in proprietary, physical data structures. These solutions leverage the semantic layer to present a more user friendly, logical view of the data on a physically separate copy of the data. The combination of specialized data visualization tools and optimized, physical data structures (i.e. OLAP cubes) yielded the “speed of thought” analytics capability that is the foundation of modern business intelligence programs.
The key limitations of traditional OLAP are 1) OLAP cubes must be loaded and frequently refreshed to stay current; 2) building OLAP cubes from big data sources or data with high dimensionality is impractical, leading to compromises in how granular analysis can get; 3) different teams may have different analysis priorities, leading them to define alternative views to support their needs. This ultimately leads to cube sprawl and competing logical definitions (i.e competing semantic layers) for different teams. In other words, there’s no single source of the truth.
Integrating Data Virtualization into a Semantic Layer for BI & Analytics
AtScale extends the original notion of a semantic layer by combining the benefits of dimensional analytics with data virtualization and an intelligent orchestration engine (that we call autonomous data engineering). The goal has been to deliver powerful analysis capabilities to data consumers, regardless of which BI platform they use (e.g. Tableau, Power BI, Excel, Looker, and more). The key benefits of this approach are:
- Self-service BI Guardrails: Analysts and data scientists request data through the semantic layer’s business-friendly logical model,connecting to AtScale with tools of their choice. The semantic layer forms a single source of truth for business metrics, as well as the dimensions on which data consumers define queries.
- Live Cloud Data Access: The semantic model presented to the user maintains a virtualized connection to underlying cloud data sources. As users request data through the semantic layer, virtualized queries pull data from the underlying source in real time. Any transformation needed to translate raw data to the logical model happens on the fly, with no need to manually transform the underlying dataset – regardless of whether the raw data is in a cloud data warehouse (like Snowflake) or a data lake platform (like Databricks).
- Query Performance Acceleration: AtScale accelerates analytics query performance in two unique ways. First, by leveraging the concept of dimensionality, the AtScale engine automatically creates aggregations through its knowledge of data relationships and query behaviors. The aggregations are defined and stored within the cloud infrastructure, leveraging an AI engine and intelligent data locality using what we call “preferred aggregate storage.” Second, a cloud orchestration engine optimizes the use of the underlying cloud data platform. This capability is key to taking full advantage of organizations’ investments into powerful modern cloud data platforms– and for managing cloud costs.
The AtScale semantic layer plugs into an organization’s data fabric, delivering a unique approach to dimensional modeling and performance acceleration for analytic use cases. While AtScale is a true dimensional analysis platform, it is differentiated from traditional OLAP approaches in that it does not rely on a physical cube. Reliance on separate, physical data structures for dimensional analysis limits the scale of data access to days or weeks. Most modern analyst teams require access to multiple years’ worth of data to make informed, data-driven decisions.
AtScale has pioneered the concept of a virtualized, dimensional query capability that delivers all the speed and analytical sophistication of traditional OLAP solutions, without the baggage and scalability limitations of creating separate, physical copies of the data.
The AtScale semantic layer is a virtualized dimensional modeling and analytics platform.
Most companies do not use just one BI tool. One team may work in Microsoft Power BI while another builds dashboards in Tableau or Looker. The issue is that teams often end up recreating the same pipelines and calculations across platforms. Data virtualization helps avoid some of that extra work by allowing different BI tools to access data through a shared layer rather than rebuilding everything separately.
They can. A company might have finance working in Microsoft Power BI while another team builds reports in Salesforce, Tableau, or Looker. Instead of maintaining separate copies of the same data for every platform, teams can point those tools to the same virtualized layer and work from a shared data and logic layer.
Most companies end up with more BI tools than they originally expected. One team adopts Microsoft Power BI, another prefers Tableau, and before long, the same data is getting rebuilt in different places. Data virtualization helps prevent that from becoming a maintenance mess by providing teams with a shared way to access data across platforms, rather than recreating the same logic over and over.
SHARE
Whitepaper | Enterprise Semantics for Power BI