Self-Service Analytics and Business Intelligence: A Myth or the Key to Escaping Report Factory Hell

Posted on 11th Aug 2020

By Rob Woollen

Self-service analytics and business intelligence may help ease the sharp divide that exists between those who speak the language of data and non-technical domain experts.

Over the last decade, the amount of data created, consumed, and stored has exploded. In an IDC white paper from 2018 predicted that the collective sum of the world’s data would grow from 33 zettabytes this year to 175ZB by 2025, for a compounded annual growth rate of 61 percent. Measuring the volume of data is an inexact science, but the predictions bear out so far. They may even fly past them in our shifted pandemic reality, where people work from home and consume even more media and interact virtually rather than in the real world.

Fueled by wearables, IoT, social media, and mobile phones, all this data is a treasure trove for companies that can use it safely and effectively. By properly parsing through structured, unstructured, and semi-structured data, companies hope to find patterns, identify emerging trends, and extract insights that are not obvious from the surface, insights that will drive better decision-making and give them an edge over the competition.

Businesses are looking to employ data engineers, data scientists, and data analysts in record numbers to carry out this sort of analysis. Data scientists are in short supply, LinkedIn’s U.S. 2020 Emerging Job Report shows that the Data Scientist role is becoming increasingly prevalent. Yet, despite the mountains of data at their fingertips, and data experts at their disposal, most organizations are still not maximizing their data and making it work for them. In fact, many organization’s data teams have descended into the depths of report factory hell. They generate numerous low-level, ad-hoc reports in an endless attempt to give their colleagues the answers they need while struggling to tackle the high-value data projects that actually move the needle for their business.

Companies need to move lightning fast, and course-correct in real time to compete. The confidence that data-driven decision making can provide is appealing to business leaders across organizations. In every team and department, people need to be able to ask questions, get answers, and keep iterating to gain insights. But this puts incredible strain on the data team. A sharp divide exists between those who speak the “language of data” and non-technical domain experts. Instead of investing in education, promoting data literacy, and laying the foundation for collaboration between these two camps, companies throw bodies and technology at the problem, setting everyone up for failure.

For over a decade now, self-service analytics and business intelligence (A&BI) have been heralded as the solution that would bridge this gap, freeing data teams to focus on their work, while giving business users the ability to get the insights they need to make better decisions. Instead of opening the door to more engagement with the data, the result has often been the opposite, an expectation that for more reports, more often. The empty promises of self-service A&BI are one reason for the imbalance between a business’s data and the insights they have been able to draw from it.

By Gartner’s definition, self-service A&BI has largely been a myth. Far from being “hands-off,” the tools that claim to offer self-service to everyday teams – marketing, procurement, sales, operations, etc. – actually require a significant amount of work from the IT department – not to mention loads of complicated SQL code to set up, use, and maintain. Days, or even weeks, of specialized training and/or daily assists from the data team are often necessary to answer even basic questions. Unsurprisingly, the adoption of these solutions hovers at a paltry 35 percent.

Insights require domain expertise, as well as data smarts
The democratization of data and community-driven A&BI are the future, but openness needs to be balanced with security, to provide controlled data exploration. While teams know their jobs well, many have modest technical abilities when measured against the kind of code and technical know-how required by many A&BI solutions. Nobody can do everything. The goal should be to enable anyone to ask any questions of the data securely and safely.

At the same time, businesses need to retain extensive security, governance, and control over data, so organizations must ensure that everything within the cloud data warehouse (CDW), and by extension within the analysis tools are safe and correct, ideally creating a single source of truth for the entire company. The following four requirements offer a path towards this goal.

• Enable everyone to explore: Flexible, yet familiar interfaces can connect domain experts directly to the CDW – no SQL required. With some basic spreadsheet formulas, teams can explore and analyze billions of rows of data in real-time without writing a single line of code.
• Focus on the work that matters: By empowering domain experts to explore the data themselves, data teams are free to pursue higher priority work, like uncovering new data sources, building new data models, and solving impactful problems so they can uncover more value for the organization as a whole.
• Maintain security and compliance: This freedom and flexibility does not come at the cost of security or control. The best next-generation A&BI tools run natively with your existing CDW, using a secure connection to query your warehouse directly. Results of queries are returned in the browser – so no data is ever stored at any time. Administrators can set permissions by team and namespace and can restrict data access directly from the database as well. Providing a single point-of-access for your data, so you can establish robust data governance, eliminate dangerous Excel extracts, and keep data off local PCs.
• Reuse analyses and collaborate with others: the best A&BI tools empowers business and data users to collaborate with one another to build more robust and contextual data models and foundational datasets. When everyone can share and build on each other’s work, the time to insight and data ROI is accelerated exponentially. The data remains accurate and up-to-the-minute because it is directly accessed from your CDW, so you never have to worry about duplication or creating redundant reports again.

This is not a pipe dream; it is possible to strike a balance between governance and freedom. CDWs and next-generation A&BI tools are making true self-service finally possible. The cloud data stack incorporates core principles that were previously restricted to software development: version control, live data access, and connected environments. By leveraging the strengths of CDWs, business leaders and domain experts can be included, even lead the data conversation, and give data teams the ability to help drive business outcomes.

Eliminating report factory hell is possible when the fog clears, and the path to true data insights is in view.