March 5, 2019
What’s the best BI tool for Hadoop?
This blog is part of a series from Tobias Zwingmann, Managing Partner at RAPYD.AI. RAPYD.AI lets you easily and quickly build fully-functional AI prototypes or AI MVPs powered by state of the art AI-services from Google, Amazon and Microsoft. Tobias has over 15 years of professional experience working in the corporate world where his responsibilities included building data science use cases, digital B2B products, and developing an enterprise-wide data strategy. He is also the author of the O’Reilly book, AI-Powered Business intelligence (2022). Follow Tobias on LinkedIn and Twitter.
—
The fields of data science and business intelligence (BI) are constantly evolving. Tools and technologies come and go all the time, and new buzzwords pop up with almost seasonal frequency. One thing that remains constant, however, is the need for data science and BI teams to collaborate effectively to gain valuable insights and make business decisions based on data.
Unfortunately, this collaboration can prove to be a difficult task.
In most organizations, data science and BI teams are in different departments, have different levels of leadership, and don’t share the same goals — making it extremely difficult to collaborate effectively and bridge the different tools and languages.
This can lead to confusion, errors, and lack of trust between teams, resulting in overall subpar performance.
In this blog post, we will discuss four key best practices for data science and BI teams to work more closely together: Aligning Language, Aligning Tools, Aligning Data Models, and Aligning Culture.
With these data science and BI best practices, organizations can overcome collaboration challenges and create a more seamless and effective data science and BI ecosystem.
Best Practice for Data Science and BI Teams #1: Align Language
One of the biggest challenges for data science and BI teams is the lack of a common language. Data Science and BI teams often use different terms and concepts, which can lead to confusion and misunderstanding. For example, a data scientist may use the term “model” to refer to a predictive machine learning model, while an BI analyst may use the term to refer to a relational or dimensional data model in an OLAP cube.
Similarly, terms like “measures”, “dimensions” or “report filters” are everyday language for BI professionals, while data scientists may argue about the right set of “training examples,” the best “predictive features,” or “sampling methods.”
The lack of a common language can lead to a lack of trust between teams and poor communication overall, which in turn leads to poor collaboration, as it can be difficult for teams to share and understand each other’s work.
To establish a common language between data science and BI teams, organizations should create a glossary of terms used by both teams. This glossary can include definitions of common terms as well as explanations of how they are used in the context of data science and BI.
This glossary should become the first point of reference for members of both teams so that they no longer have to secretly Google and guess what the other party might mean. In addition, cross-functional training sessions can be held to ensure that both teams have a clear understanding of the terms and concepts used by the other team.
Furthermore, having clear documentation and procedures in place, where the team can refer to, can also help to align the language. This documentation may include data dictionaries, data lineage, data governance policies and procedures, and other important documents that teams may need to refer to.
In summary, by aligning the language, organizations can ensure that both data science and BI teams have a common understanding of the terms and concepts used in the field. This leads to better communication, faster problem solving, and better collaboration between teams. It also helps maintain clear documentation and procedures, which is important for data governance and regulatory compliance.
Best Practice for Data Science and BI Teams #2: Align Tools
Another challenge faced by data science and BI teams is the use of different tools and technologies. Data Scientists often use programming languages such as Python or R, while BI analysts typically use tools such as Tableau and Power BI and the ETL stack built around them. These different tools can make collaboration difficult, as it can be challenging for teams to understand and share each other’s work.
For example, a data scientist may use Python to build a predictive model, but the BI team may have difficulty understanding what the code is doing and how to interpret the results of the model — let alone integrate it into their ETL pipeline.
On the other hand, a BI team may use Power BI to build a dashboard. Meanwhile, the data scientist may not be able to access the transformations and KPI calculations that are done there, so they must painstakingly reconstruct the results in Python and verify that they’re getting the same results shown in the BI dashboards.
To overcome this challenge, organizations can encourage the use of common tools for both teams — each for their specific tasks. For example, data scientists could be trained in BI tools to use their data visualization capabilities, and BI analysts could be trained to read (and perhaps even write) Python or R code. An even more elegant solution would be for data science and BI teams to commit to using a common technology as much as possibleOftentimes, that’s SQL.
While it may be more tedious for a Data Scientist to write a data preparation pipeline in SQL than in pandas, the advantage is that BI teams can easily use this pipeline. This not only increases efficiency and improves data quality, but also allows both teams to understand and interpret each other’s work.
Needless to say, having a common data platform for data storage and management is an important catalyst. By using a common platform, teams can access the same data and collaborate more easily. This also helps with data governance and security compliance.
By aligning tools, organizations can ensure that both the data science and BI teams are working with the same set of tools and technologies. This leads to greater efficiency, better data quality, and better collaboration between teams. It also helps with data governance and security, as well as faster problem resolution and better decision making.
Practice for Data Science and BI Teams #3: Align Data Models
A third challenge facing data science and BI teams is the lack of alignment between data models. Data scientists typically require data in a denormalized, granular form, while BI analysts typically require normalized, dimensional, and relational models. These different models can lead to confusion and errors when teams share data.
For example, a data scientist might use a complex model to predict customer behavior but might use a different logic to create critical business entities, such as defining customers, calculating their revenue, or assigning customer-specific attributes. These different definitions can create a lot of confusion and distrust among teams — and ultimately lead to poor results.
To overcome this challenge, companies should use centralized, well-managed feature stores for models used in production that serve as a source of truth for both the data science and BI teams. A data dictionary should also be created to document the data models used by both teams, including the purpose of the model and the appropriate owners who can answer further questions. This will ensure that both teams have a clear understanding of the data models and how they are being used.
In addition, joint data modeling sessions can be held to ensure that both teams have a clear understanding of the data. In these sessions, data scientists and BI analysts can work together to discuss the data, identify any gaps or inconsistencies, and develop a common understanding of the data. This ensures that both teams are working with the same data and that the insights gained are accurate and actionable.
By aligning data models, organizations can ensure that the data science and BI teams are working with the same business logic and produce consistent insights. This leads to a more efficient and effective data science and BI ecosystem that ultimately creates more value, making it easier to achieve business goals.
Best Practice for Data Science and BI Teams #4: Align Culture
The cultural differences between data science and BI team members are the last major piece of the puzzle that hinders effective collaboration. Data scientists and BI analysts often have different communication styles, problem-solving approaches, and ways of working. These cultural differences can lead to misunderstandings and delays in problem solving.
For example, data scientists have a more experimental and research-oriented approach, while BI Analysts have a more regimented and business-oriented approachThis is due to their different backgrounds.
BI teams and processes have typically evolved over decades of working with data warehouses and well-managed reports. Data science teams, on the other hand, have often been hired with a more agile mindset, commonly incubated in “data labs” trying to explore and push the boundaries of what’s possible with data. This can lead to different priorities and a lack of understanding of each other’s perspectives.
To align cultures, companies should encourage cross-functional project teams and organize team-building events. The best way to better understand each other is to work on projects together. Then, data science and BI teams can develop a better understanding of each other’s roles and responsibilities and learn to appreciate the value each team brings to the business.
Regular communication and knowledge sharing sessions are also very helpful. This will give a space for the data science and BI teams to share their insights, experiences, and best practices. This can help break down barriers and build trust between teams.
By aligning cultures, organizations can ensure that the data science and BI teams work together more effectively and contribute to an overall more data-centric and data-driven culture across the organization.
Getting Data Science and BI Teams in Sync
In this blog post, we discussed four key data science and BI best practices for bringing teams closer together and producing more value: aligning language, aligning tools, aligning data models, and aligning culture:
Align language: Ensure a common understanding of the terms and concepts used in the field.
- Align tools: Ensure that data science and BI teams work together with high efficiency.
- Align data models: Ensure that data science and BI teams both produce insights that are accurate and actionable.
- Align culture: Increase the effectiveness of data science and BI teams and make sure their combined output is larger than the sum of its parts.
Following these best practices for data science and BI teams, organizations can overcome the often historically induced challenges of working together and create a more seamless and effective business ecosystem.
In conclusion, aligning language, tools, data models, and culture will help organizations create a more seamless and effective data science and BI collaboration, which will ultimately produce more value for the organization.
After all, when the data departments are not working together effectively among themselves, who else will?
That’s why these best practices for data science and BI teams should be on top of your data agenda!
NEW BOOK