A Proof-of-Concept of BigQuery
Join the DZone community and get the full member experience.Join For Free
04 september 2012
google recently upgraded its enterprise offerings with an improved version of google app engine and some new tooling. google bigquery is the new online service for running interactive queries over vast amounts of data—up to billions of rows—with great speed. it’s good for analyzing large quantities of data quickly, but not for modifying it. in data analysis terms, bigquery is an olap (online analytical processing) system, aimed at helping organisations work with big data.
we were very interested in putting this technology to the test, so we searched for a partner with a data set worthy of the label "big". one of our clients in the uk, autotrader, had both big data and a business issue that they hoped to see if bigquery could help them with.
autotrader is the top-ranked online site to buy and sell new and used cars in the uk, and also publishes a weekly classified ad magazine. two questions a dealer wants to know the answer to when a used car comes onto the lot are: how much to offer for it; and how long it might take to sell it. the answers lie within autotrader’s historical data.
we borrowed data from autotrader covering all ads placed on their website over the last five years, including how long the ad was displayed and the price of the vehicle when it was removed from the site (the assumption being that was the selling price of the vehicle). the test data set contained over 750 million rows, and had been uploaded to amazon’s ec2.
getting the data into bigquery
dealing with big data, file size limits are an issue. we used the unix command line 'split' to break the data file into chunks of the right size, taking care to break files apart on line boundaries rather than the middle of a record. both google cloud storage (gcs) limits and the bigquery import limits have to be considered when doing this.
as soon as we had google’s gcs command line tool gsutil successfully installed, we used it to move the data from amazon into gcs, which turned out to be the slowest part of the whole process. the total rows took around 12 hours to transfer to gcs. we tried using the parallel upload facility in gsutil but this just led to our ec2 instance becoming i/o bound. with hindsight, we could have spent more time experimenting with amazon regions, file sizes and parallel uploads to make this process quicker.
with the data in gcs, we next created a very simple text file to represent the schema and used it with the big query command line tool ( bq ) to set up tables in bigquery. we checked it out first with a small subset of the data, doing a few queries from the bigquery web console to be sure everything was suitable before we loaded the whole dataset.
the next stage was to complete the transfer from gcs into the newly defined bigquery database. because of the problems we ran into with parallel uploads into gcs from our source on amazon, we didn’t try to use the parallel facility for the load into bigquery. later we found we could have done this and probably cut our 8 hour transfer time down to around 15 minutes.
once we had all the data uploaded we tried a few queries. at the start, these were very slow, perhaps to do with some internal distributions not having happened inside of bigquery. initially these queries were taking a few minutes, but the next morning things took around 7 to 10 seconds and this remained reasonably consistent for the remainder of the exercise.
creating big data web analytics using bigquery and google charts
our web pages made restful calls to our app which in turn used the java big query api (python is an option as well) to make restful calls to invoke queries on the data. the results were then rendered client-side using the google charts libraries. in addition to the interactive charts, we were able to add a simple export mechanism, taking advantage of the fact that bigquery results are saved into a temporary table which can be accessed via a "job id". this makes for a very simple and quick way to export the results of queries into gcs—and hence into other google tools or as a csv file. all of this was secured via oauth giving us fine-grained control over who could see, access and invoke queries using the app.
we began looking into changing our web app to use asynchronous mechanisms for invoking bigquery, but ran out of time. that said, it looked pretty straightforward, using the jobid to query for intermediate results. given the chance again, we would go with an asynchronous approach from start.
conclusions and benefits
overall we were impressed with what we were able to achieve in a short amount of time with bigquery, the apis and utilities. we were able to experiment with different ways of visualising and querying a very large volume of data with a relatively low investment in terms of time and without needing expensive infrastructure.
for organisations longing to gain insights into very large datasets, bigquery could be a viable option that requires no purchases of specialised software or additional infrastructure. even for organisations that already have an enterprise data warehouse, bigquery is an option for testing out theories, or for other instances where the keepers of the data warehouse place restrictions on use.
issues and lessons
we encountered issues during the proof of concept that others can learn from.
installing the google utilities
make sure the right version of python is installed, and multiple versions are not on the path, to avoid issues. we found it best to use the downloadable versions of the tools, as our early attempt, trying to use easy_install with an older version of python caused a lot of problems.
google codebase examples
it proved actually quite difficult to find some working java examples of bigquery used from within app engine, especially around the oauth mechanisms. but once we'd created a few classes to handle the work, we had no further issues, even with a four-stage redirect (as thoughtworks uses its own corporate oauth mechanism with google apps). some more thought would be needed to make these mechanisms work with something like webdriver and automated acceptance tests.
charts and bigquery have slightly different json formats
another issue is that the json format returned from bigquery is slightly different from the one charts expects. while there are good reasons for this, having charts able to directly consume some of the json would have cut the amount of code we had to create.
timestamps were a problem
our data included timestamps in a non-standard format (from the google tool’s perspective). this made it difficult to construct queries that selected by time periods. while we were able to work around this, it did limit some of what we'd hoped to do with the data. we've passed this along to the bigquery team and expect they'll have some more flexibility around date/time formats in the future.
use the web console
while experimenting we found the bigquery web console invaluable for trying out queries before adding them into the code.
experiment with gsutil and bq parallel loading
while we encountered i/o issues loading from amazon to gsc, we later found we could have saved many hours by using the parallel load from gcs into bigquery itself. it is worth testing the various options available with gsutil and bq to see which give the best performance for your environment.
watch your data usage as you execute queries and develop with small datasets
if you are doing a lot of testing of a full dataset, you can get into several or more terabytes of data usage, which can generate a high charge. queries come back so fast, you forget that return sets can be many gigabytes—and they add up fast.
Published at DZone with permission of Martin Fowler, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.