Choosing The Right Technology

Enterprises face challenges of overcoming the limitations of existing legacy technology and providing the performance necessary to drill into data at scale. The full analytics stack relies on three components:

• A data warehouse that can support the capacity demands of the business
• A modeling platform to provide consistent data definitions analysts can use to drill into data
• A visualization tool to derive the insights that are ultimately used to make business decisions.

The first step in choosing the right technology is to establish the goals of your organization and answering the question: What are the business outcomes that you’re trying to achieve? You may be focused on being data-driven at scale to leverage rapidly growing data volumes, reducing reliance on IT in order to accelerate reporting and time-to-insight, or enabling granular drill-down analysis capabilities to better identify business opportunities – or a combination of these.

With your goals established, it’s important to define your technology evaluation criteria. Criteria should drive performance and ultimately our business goals. Examples of these are the capacity of the platform, usage of the platform, and the compute cost of the dashboard or data model.

You should be able to measure the results of your technology choices as well, using KPIs such as the number of data requests being managed across teams, query performance, and number of data sources being accessed regularly.

Monitoring Technology and Usage

Adopting the right cloud technology offers a tremendous opportunity for both cost savings and performance scalability. However, if the technology is used without oversight, there is a very good chance that performance expectations will not be met and unpredictable costs will erase any of the cloud’s value. This is why it’s incredibly important to implement a monitoring framework to get (and keep) your BI stack in shape.

Performance bottlenecks occur when resources exceed thresholds at peak loads and user concurrency results in queuing. To identify resource utilization bottlenecks you can set up very detailed, real-time monitoring of our systems. Metrics to track include CPU, memory, disk I/O, network traffic, and query response times.

One of the most common bottlenecks is user request queues. This can be overcome with small configuration changes in the data platform. In cases where tuning of the existing environment isn’t enough, the next option is to scale horizontally with more machines or vertically with more powerful machines. This is always the second option though, as scaling machines is never free!

Without this depth of monitoring, costs can quickly get out of control. Cloud platforms have different pricing models and you need to optimize for your situation. For example, on-demand pricing allows a customer to pay as they go based on the amount of data processed, while flat rate pricing has a fixed fee for guaranteed processing capacity. Other pricing models charge based on data volume or size and number of warehouses and clusters.

Closely monitoring usage will help you understand which pricing model will work best for you. Once you understand how users are querying data, you may decide that fixed rate pricing works better for you than on-demand, or vice versa.

You can get as nuanced as establishing fixed capacity for most of the time and paying for flex-capacity during predictable peak usage times. For example, fixed capacity may serve you well during most of the week but shifting to flex-capacity during times where processing demand increases regularly, like on Mondays when the business is updating reports and dashboards. This helps you avoid overpaying for larger fixed-capacity that won’t be used the majority of the time.

Continuously Improve Your Environment

With the right technology and the proper monitoring in place, it’s time to improve outcomes of the investment. Improvement should be a never-ending process. There are a number of initiatives that can make a world of difference for performance such as adjusting filter settings in a dashboard, updating a data model, improving data preparation, and code rewriting. Fortunately, you will find the answers for where to focus in the logs.

The logs are a record of what users are experiencing and the impact those experiences have on a technical environment. To improve return on investment, it’s important to map the logs to the drivers of performance and cost so that you can optimize KPIs like compute costs, query performance, and cost per query.

The easiest way to interpret logs is by visualization. An efficient way to do this is to export all of our logs, load them into your data platform, and query the logs for analysis. You can then visualize that analysis in meaningful depictions like box plots and scatter plots to help identify areas of improvement. One caveat: be careful about using averages as they don’t provide a good depiction of performance.