The Company and the Challenge
Modern financial technology (FinTech) companies are banking on data analytics solutions to fuel their future growth. The global FinTech market was valued at $110.57 billion in 2020, and is projected to grow to nearly $700 billion by 2030 (GDS Link). Competition is fierce, so forward-looking FinTech companies are seeking to use data to improve user experiences and uncover new revenue opportunities.
Trumid is one such company. A Forbes Fortune 50 FinTech leader, Trumid is working to move the corporate bonds market to digital platforms as much as possible. The company’s platform currently handles about $2 billion in transactions per day. Behind the scenes, its data science team operates three use cases: reporting, analytics, and machine learning. Specifically, it’s seeking to automate reports to stakeholders, allow end users to conduct analytics inquiries, and uncover future uses for machine learning.
Due to latency that resulted from pre-calculated extracts in their previous data and analytics technology stack, stakeholders were unable to use data to inform same-day decisions. Furthermore, each time someone wanted to ask a question, they had to ask the data team to build a model. That friction meant it took a lot longer to answer questions. Finally, they needed a way to ensure they were reporting consistent numbers. These challenges led them to build a new data and analytics stack to address their need for self service Looker analytics in near real time.
The Solution:
As the team built out its new data and analytics tech stack, they selected BigQuery for their data warehouse and Looker for the BI consumption layer. To allow for self service BI and reduce latency, they needed to separate visualizations in Looker from the underlying tables in BigQuery. They wanted to expose data models to users without requiring them to understand the joins, filters, and aggregations involved in BigQuery. Furthermore they needed to reduce latency to allow for real time analysis – or as close to real time as possible.
With these considerations in mind, Trumid’s team chose AtScale to address their modeling challenges. AtScale is a universal semantic layer that sits between BI tools and cloud data platforms, enabling data science teams to centralize control and governance while decentralizing the creation of data products.
For AtScale, speed is a key principle. AtScale boosts performance in the data warehouse by intelligently pre-aggregating data. This ultimately reduces latency at query time. Self service is another guiding principle: the semantic layer allows data engineers to define models for data then expose them to stakeholders in familiar business language. Since these models are made available to any user with the appropriate privileges, the semantic layer ensures that data is consistent across teams and reports.
Finally, AtScale’s out of the box integrations with the most widely used BI tools and cloud platforms allows its customers to improve performance with minimal work required from data engineers.
The Outcome
AtScale empowered Trumid’s data engineering team to accomplish the following:
- Reduce latency from one day to around one hour, which meant stakeholders could inform decisions with same-day data;
- Separate tables in BigQuery from visualizations in Looker, so non-technical users do not need to understand the joins, filters, and aggregations involved in a table;
- Make data available so anyone can create dashboards, reports, or ad hoc visualizations;
- Report consistent numbers to internal and external stakeholders.
Furthermore, the way AtScale syncs with Looker and BigQuery helps Trumid get more out of these platforms. For example, Looker’s dynamic permissions model allows Trumid to feel comfortable granting people access to Looker, ensuring those users see the same data as everyone else while complying with regulatory requirements. Pairing this with AtScale means any user is able to create visualizations and make data-driven decisions. And thanks to AtScale, users are able to do this without any help in terms of creating models, asking permissions, or sharing.
Overall, AtScale saved the Trumid data engineering team a significant amount of repetitive work around building dashboards and ETL pipelines. Now they are free to work on machine learning research to identify new ways to apply intelligence or automation in their platform.
“AtScale and Looker run themselves these days. AtScale made it so seamless we don’t have to spend much time on maintenance, it’s just adding features we have to spend time on.”
-Chris Reid, Senior Data Engineer