Over a million developers have joined DZone.

Making Big Data Analysis Accessible Throughout Your Company

DZone's Guide to

Making Big Data Analysis Accessible Throughout Your Company

Andy Granowitz, an engineer at Wagon, shares how to make SQL more accessible to those who may not be familiar with it and how that could increase productivity.

· Big Data Zone ·
Free Resource

The open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.

Thanks to the likes of Google Analytics, Mixpanel, and others, it's easy to measure app engagement. The problem is these services only report on a predefined set of metrics and user segments. Also, it's hard to analyze other customer interactions, such as sales calls or support interactions, alongside app usage data. When you need more custom analysis or to combine product usage with other data sources, you'll need to query the raw data. I'm talking user IDs and timestamps.

There are a number of great modern ETL tools that centralize app engagement and other customer interaction data into a data warehouse like Redshift or BigQuery. Having all of your analytics data in Redshift and queryable by SQL is very powerful: It allows you to answer nearly any question you have of your data.

But there's a Big Problem™: Only some people know how to write SQL. Engineers and savvy analysts will be comfortable writing SQL, but less technical members of your team also need access to the data. What's the best way to make everyone at your company data driven, regardless of their SQL knowledge?

We've found three ways to to help everyone at your company benefit from SQL.

Make It Easy and Fun to Share SQL

Sharing queries can be an important, rewarding part of your team's culture.

At Wagon, we encourage this in two ways. We have a dedicated Slack channel for posting ad-hoc queries, data, and charts. Slack is a great place to show off your hard work! We also keep a repository of "source of truth" queries that are code reviewed, commented, and frequently updated. Having a place for ad-hoc sharing and more standard reports solved a lot of confusion.

Wagon SQL shared on Slack

Reuse and Modify Your Teammates' Work

Many questions can be answered with the same underlying SQL. Make it easier to browse other people's queries by having them in one place and with reasonable comments. We see a lot of people tweaking a query by changing a where clause to use a different customer ID or log line. If you're using data from another analytics service, many provide starter queries. Here are a few: Amplitude, Google Analytics, Segment, and Snowplow.

Learn some SQL

SQL isn't that hard to learn. We joke that you can double your salary by learning how JOIN works. We recommend Udemy's online SQL course or if you prefer in person, check out General Assembly's SQL bootcamp.

Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.

big data ,sql ,slack ,communication

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}