Picking colors for a black-and-white pictures is an easy and natural task for people. Some things are easy to figure out like grass and skies, and some are indeterminate e.g. what color was that shirt? Or car? But we can make a pleasant enough assumption.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich who are Researchers from the Toyota Technological Institute at Chicago and University of Chicago developed a fully automatic image colorization system. And they did it using off-the-shelf GPUs that anyone can buy. The key advancement in their technique is that it requires no human guidance during the process. In some sense it has learned the relationship between grayscale distributions and color distributions by being trained on a large number of "black-and-white" and "full-color" image pairs.
Using a conventional Nvidia TITAN X GPU, they trained their deep neural network to predict hue and chroma distributions for each pixel based on what they call a hypercolumn descriptor. Essentially they substituted the predicted color distributions at each pixel for the original grayscale distributions.
Above are actual examples using a fully automatic algorithm. The left column is a grayscale version of an image that was used for input and the right column is the full-color representation of reality. The researchers considered that "ground truth". The middle column was generated solely from the grayscale image to its left.
Clearly there are many places that we can take advantage of this automatic colorization technology (e.g. historical photos and old motion pictures). Using such inexpensive high-performance hardware for the learning process and of course much less power for the rendering process it seems likely that we will see commercial versions soon. How long will it be before it's a standard feature in Photoshop or Final Cut Pro?
Another thing worth mentioning is that this technique leads to far more natural looking colorization (see the examples below). I think it's very unlikely that you would have even suspected that these images were colorized without any a priori knowledge of the colors in the original scene.
While the above photos are rendered with believable colors there are a number of situations where color selection will have to be more imaginative. Certainly the colors below are believable ... But what is real? Note: the image on the far right represents the distribution that correlates with saturation, but the hue was indeterminate.