TensorFlow Without the Server Farm?
TensorFlow Without the Server Farm?
The weighty consequences of a bright idea? Learned TensorFlow models can now be executed on small devices. TensorFlowLite is part of a natural progression. It's worth a look!
Join the DZone community and get the full member experience.Join For Free
Insight for I&O leaders on deploying AIOps platforms to enhance performance monitoring today. Read the Guide.
Google's recent and significant contribution to the machine learning universe is TensorFlow. It's been adopted by a large number of groups ranging from academia through completely commercial, customer-facing applications. Anyone who understands what it does has always understood that it takes massive computational resources and was designed for massive data sets. Owning an entire data center was a good place to start if you wanted to play with TensorFlow. Well, that's all changed now.
TensorFlow Lite is targeted at mobile devices and even small embedded devices. While the problems it will address are less sophisticated, it will address a problem everyone recognizes: latency. The world seems awash with microservices — and don't get me wrong, they provide a great variety of interesting functionality — but they all live in the cloud. Even worse than being slightly unresponsive is that they are unpredictably variable when it comes to round-trip communications. A one, two, or more second-time delay penalty is added to every interaction. So, even if the microservices computation is instantaneous it still seems sluggish. We've all gotten used to the fact that using speech on our mobile device is not at all like talking to a person. It's more like a walkie-talkie: "What's on my schedule at 11 AM? Over." "You have a conference call with Bill. Over."
TensorFlow Lite will help greatly with services where all the modules are simple enough to execute on the device. Incorporating more of these device-resident intelligent microservices will lead to more snap and sizzle in the interactions.
What makes this a "lite" version of TensorFlow is the elimination of the training phase. TensorFlow lite excepts trained models from the full-blown TensorFlow system as input and translates them into significantly lighter weight models that are optimized for maximum execution speed at runtime.
First, you need to create a trained TensorFlow model and save it as a file. Second, you process the model via the TensorFlow Lite converter, which generates the smaller faster version (based on FlatBuffers) for execution.
When you're ready to deploy you upload this more compact file to your mobile app and access it by either a Java or a C++ API. Either of those APIs invokes the TensorFlow interpreter, which operates on the input parameters to generate a result. Google says that even with all of the potential operators loaded, the footprint is still only about 300 KB! And if that's not enough, recent Android devices with hardware acceleration hardware will automatically take advantage of the Android neural networks API for even faster performance.
There are several excellent models that are available to play with without having to get into the nitty-gritty of training your own:
- MobileNet: A class of vision models able to identify across 1000 different object classes, specifically designed for efficient execution on mobile and embedded devices
- Inception v3: An image recognition model, similar in functionality to MobileNet, that offers higher accuracy but also has a larger size
- Smart Reply: An on-device conversational model that provides one-touch replies to incoming conversational chat messages. First-party and third-party messaging apps use this feature on Android Wear.
And when you're ready to get more advanced, you can even retrain these models with your own data via transfer learning.
Note: TensorFlow Lite is actually an evolutionary extension of a lesser-known version called TensorFlow Mobile. While it is definitely an improvement when it comes to speed and size requirements, TensorFlow Lite is a generalization of TensorFlow Mobile which includes the world of embedded and IOT devices. Also, this is a new and fairly large R&D development. TensorFlow Lite is currently a developer preview, but I expect we will see a lot of interesting new features and functionality in the months to come!
Now is the time to start.
Opinions expressed by DZone contributors are their own.