Pentaho Data Integration Adds a Greenplum Bulkloader
Join the DZone community and get the full member experience.Join For Free
with our ongoing commitment in supporting “big data”, we at pentaho have beefed up our data integration tool by adding a bulk loading step to support greenplum, the massively parallel processing database from emc. the new “greenplum load” step provides a graphical user interface to greenplum's “gpload” utility. the “gpload” utility provides an interface to the database's external table parallel loading feature.
i am going to show how to use gpload from the command line and then from pentaho data integration. running gpload from the command line is straightforward. here i've placed the “customers-100.txt” sample file we distribute with pdi into a folder on our greenplum server:
[gpadmin@greenplum test]$ ls -l
-rw-r--r-- 1 gpadmin gpadmin 556 aug 3 20:31 control.cfg
-rw-r--r-- 1 gpadmin gpadmin 8754 aug 3 20:24 customers-100.txt
along with the data file is a control file. the contents:
- format: text
- delimiter: ';'
- quote: ''
- header: false
- error_limit: 50
- error_table: customer_100_load_errors
- table: customers_100
- mode: insert
a configuration file follows the yaml standard. from wikipedia, “yaml is a human-readable data serialization format.” wikipedia can describe it a bit more. the greenplum admin guide has excellent examples.
to load the customers_100 table i simply execute gpload:
[gpadmin@greenplum test]$ gpload -f control.cfg
2011-08-03 20:59:10|info|gpload session started 2011-08-03 20:59:10
2011-08-03 20:59:10|info|started gpfdist -p 8000 -p 9000 -f "customers-100.txt" -t 30
2011-08-03 20:59:10|warn|1 bad row
2011-08-03 20:59:10|info|running time: 0.20 seconds
2011-08-03 20:59:10|info|rows inserted = 100
2011-08-03 20:59:10|info|rows updated = 0
2011-08-03 20:59:10|info|data formatting errors = 0
2011-08-03 20:59:10|info|gpload succeeded with warnings
we did have one error: our data file has a header row and i did not remove it. that's ok, all the other rows made it into the table.
so, to get all that done i created the table in the database using db visualizer, created the yaml file using gedit and then ran gpload a few times to iron out some data mapping issues. i shared the final successful run with you.
no, it didn't really go that way, but that is a realistic scenario for database and etl developers when using bulk loading from the command line. i'll fill you in on how i set up the command line sample a little later on.
now i'll bulkload with pentaho data integration and the “greenplum load” step. i'll fire up spoon, the graphical user interface:
displayed is a two step transformation i created. the “read customers 100” step will read the customers-100.txt file and parse the data into fields. the data that read from the files is passed to the “greenplum load” step that will:
- create a temporary data file from the input stream
- write the yaml control file based on the fields in the input stream and the step's configuration
- execute gpload in the same manner as i showed in the command line example
here is the new “greenplum load” step dialog with its “field” pane enabled:
the list of fields was populated by clicking “get fields”. there's no need to enter this manually as the step can read it from the input stream provided by the “read customers 100” step.
to create the “customers_100” table (if it does not exist) just click the “sql” button to display the ddl in a dialog. clicking “execute” will create the table.
on to the “gp configuration” tab:
here i specify where the step can find the gpload program, where to write the control and data files, and what to place in the control file.
and now back to the entire transformation for an explanation on how it is executed.
we have a centos server running greenplum 220.127.116.11. it is on this server that gpload is found. being a “backend database” server we do not run the client tool, spoon, from that server. we do have a distribution of pdi on that server so we can run a carte server. this server will run the transformation.
i specified “gp8084” as the carte server to use and clicked launch to execute the transformation.
the result of the execution:
we do not have gpload throwing out the header row as we did in the command line example. the “reed customers 100” step was configured to ignore the first row of the data file.
i admit it: i did not use gedit to create the control file for the command line example. i configured the step to be “manual load (only creation of files)”. this generated yaml. while i was there i generated the table using the sql button.
thanks for reading! i intend to be back with more on the parallel load features of gpload and how clustering in pentaho data integration can aid in parallel loading.
Opinions expressed by DZone contributors are their own.