Parallel Grid Search in H2O

DZone 's Guide to

Parallel Grid Search in H2O

In this article, learn more about Parallel grid search in H2O.

· AI Zone ·
Free Resource
H2O is, at its core, a platform for distributed, in-memory computing. On top of the distributed computation platform, machine learning algorithms are implemented. At H2O, we design every operation, be it data transformation, training of machine learning models, or even parsing to utilize the distributed computation model. In order to work with big data fast, it’s necessary.

However, a single operation usually can not utilize clusters' computational resources to the very maximum. Data needs to be distributed across the cluster, and many operations require sequential execution of tasks, which, even if implemented in a distributed manner, follow after each other and require data exchange. These and many other smaller factors, if summed up together, may introduce a significant overhead.

One area of interest is the process of machine learning model training. It takes considerably longer for a machine-learning algorithm to build a model. And depending on the algorithm itself, there are inefficiencies to be found. Therefore, introducing concurrency to model training might improve the utilization of the cluster’s resources, mainly CPUs and/or GPUs.

You may also want to read:  Machine Learning With H2O — Hands-On Guide for Data Scientists

One area where lots of models are built is grid search. Grid Search “walks” through a space of hyperparameters and builds the respective models. The walking strategy might differ, but they all share the same problem — there are many models to be built. Typically, the build process is constrained by time. And the more models are built, the better.

Until now, H2O, just like other machine learning platforms, has been training one model at a time. Since H2O version, we’ve given our users a way of building models using grid search in parallel. This effectively means there may be n > 1 models trained on the cluster. While one model is waiting or doing less optimizable calculations, other models can meanwhile utilize cluster resources. This leads to more models built in less time.

How to Use Parallel Grid Search

By default, parallel model building for grid search is turned off. This might change in the near future. We’ve introduced a new parameter named parallelism for grid search. Such a parameter, available in Flow, Python, and R, is set to parallelism = 1 by default. This implies models are built sequentially while performing grid search. In order to build models in parallel using a grid search, users have two options:

  1. Set parallelism = n where n > 1,
  3. Set parallelism = 0.

Setting the parallelism argument to any number greater than 1 instructs H2O to keep exactly that many models being built at the same time until other constraints are hit (time, maximum number of models, etc.). However, setting it to 0 means H2O is free to use internal heuristics to determine the best amount of models to be trained at one time. We call this the adaptive mode.

Currently, the adaptive mode has no strict contract and the internal heuristics might change. The adaptive mode assumes the whole cluster consists of machines with equal resources available and simply trains twice more models than the number of CPUs available per node. Given a cluster of n nodes, where each node has c number of CPUs, the number of model would be 2 * c. The number of nodes or total number of CPUs available plays no role.


Parallel grid search can be enabled by simply adding parallelism = n to H2OGridSearch constructor arguments, where n = 0 for adaptive mode or n > 1 for precise control of parallelism. Specifying n = 1 means sequantial mode.



Parallel grid search can be enabled by simply adding parallelism = n to h2o.grid function arguments, where n = 0 for adaptive mode or n > 1 for precise control of parallelism. Specifying n = 1 means sequantial mode.



There is an H2O Parallel Grid Search Benchmark available on GitHub. It contains a description of the experiment as well as a script for reproducibility.

Limitations and Final Thoughts

Building lots of models in memory comes at a cost — memory. Every machine learning algorithm requires a certain amount of RAM to operate. Building multiple models implies linear or near-linear growth in memory consumption. From our observations, including the ones in the benchmark, training multiple models at one time almost always comes with a significant performance boost. It’s definitely not slower.

The smaller (relatively to the cluster’s resources) the models are and the faster to build the models are, the more effective parallel grid search is. For super-heavy models (again, this is relative to the cluster’s resources), where even training one model at a time barely fits into the memory, using the default parallelism = 1 might be the only way to train the model. In all other cases, using parallel grid search brings a potentially significant advantage.

Even using at least parallelism = 2 may bring significant speedups. Memory is the constraint. However, training too many models might introduce large context-switching overhead, which is also unwanted. The current rule for adaptive mode, which spawns 2 * number of leader node's CPUs, is empirically set for large hyperparameter space, where the model training phase is relatively short. Users are expected to experiment with this number. One of the improvements already planned is to bring a solid adaptive regime that can manage/adapt to the cluster’s resources on the fly. Stay tuned!

The following people were also involved in the parallel grid search functionality and deserve credit: Michal KůrkaJan Štěrba, Sébastien Poirier. H2O.ai is open source and can be found on GitHub

Further Reading

Anomaly Detection With Isolation Forests Using H2O

Building a GBM Model in H2O With Grid Search and Hyperparameters in Scala

h2o.ai ,machine learning ,artifical intelligence ,automl ,parallel grid search ,python ,r

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}