DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workkloads.

Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Predicting Diabetes Types: A Deep Learning Approach
  • Integrating Data Engineering Into Artificial Intelligence
  • Neural Networks: From Perceptrons to Deep Learning
  • The One-Pixel Threat: How Minuscule Changes Can Fool Deep Learning Systems

Trending

  • Kubeflow: Driving Scalable and Intelligent Machine Learning Systems
  • *You* Can Shape Trend Reports: Join DZone's Software Supply Chain Security Research
  • The Cypress Edge: Next-Level Testing Strategies for React Developers
  • Zero Trust for AWS NLBs: Why It Matters and How to Do It
  1. DZone
  2. Data Engineering
  3. AI/ML
  4. Feature Engineering for Deep Learning

Feature Engineering for Deep Learning

Many DL neural networks contain hard-coded data processing, along with feature extraction and engineering. They may require less of these than other ML algorithms, but they still require *some*.

By 
Jean Francois Puget user avatar
Jean Francois Puget
·
Nov. 09, 17 · Opinion
Likes (5)
Comment
Save
Tweet
Share
22.6K Views

Join the DZone community and get the full member experience.

Join For Free

Feature engineering and feature extraction are key — and time-consuming — parts of the machine learning workflow. They are about transforming training data and augmenting it with additional features in order to make machine learning algorithms more effective. Deep learning is changing that, according to its promoters. With deep learning, one can start with raw data, as features will be automatically created by the neural network when it learns. For instance, see this excerpt from Deep Learning and Feature Engineering:

The feature engineering approach was the dominant approach till recently, when deep learning techniques started demonstrating recognition performance better than the carefully crafted feature detectors. Deep learning shifts the burden of feature design also to the underlying learning system along with classification learning typical of earlier multiple layer neural network learning. From this perspective, a deep learning system is a fully trainable system beginning from raw input, for example image pixels, to the final output of recognized objects.

As usual with bold statements, this is both true and false. In the case of image recognition, it is true that lots of feature extraction became obsolete with deep learning. Same for natural language processing where the use of recurrent neural networks made a lot of feature engineering obsolete too. No one can challenge that.

But this does not mean that data preprocessing, feature extraction, and feature engineering are totally irrelevant when one uses deep learning.

Let me take an example for the sake of clarity, taken from recommending music on Spotify with deep learning. I recommend reading this article as it introduces deep learning and how it is used in a particular case pretty well.

I will not explain what deep learning is in general. Suffice to say that deep learning is most often implemented via a multi-layer neural network. An example of a neural network is given in the above article:

Data flows from left to right. The input layer, the leftmost layer, receives an encoding of songs. The next three layers are max-pooling layers. The next layer computes mean, max, and L2 norm of its input data. The next three layers are convolutional layers, and the last layer is a temporal pooling layer.

Don't worry if you don't fully understand what all this means. The key point is that learning only happens between the three convolutional layers. All the other layers are hard-coded feature extraction and hard-coded feature engineering. Let me explain why:

  • The input data is not raw sound data. The input data is a spectrogram representation of the sound obtained via a Fourier transform. That data transformation happens outside the neural network. This is a drastic departure from the above quote that claims deep learning starts with raw data.
  • The next three levels are max pooling. They rescale their input data to smaller dimension data. This is hard-coded, and it is not modified by learning.
  • The next level computes the mean, the max, and the L2 norm of time series. This is a typical feature engineering step. Again, this is hard-coded and not modified by learning.
  • The next three levels are convolutional levels. Learning happens on the connections between the first two convolutional levels and on the connections between the last two convolutional levels.
  • The last level computes statistics on the data output by the last convolutional level. This is also a typical feature engineering. It is also hard-coded and not modified by learning.

This example is a neural network where most of the network is some hard-coded feature engineering or some hard-coded feature extraction. I write hard-coded as these are not learned by the system, they are predefined by the network designer, the author of the article. When that network learns, it adjusts weights between its convolutional layers, but it does not modify the other arcs in the network. Learning only happens for 2 pairs of layers, while our neural network has seven pairs of consecutive layers.

This example is not an exception. The need for data preprocessing and feature engineering to improve the performance of deep learning is not uncommon. Even for image recognition, where the first deep learning success happened, data preprocessing can be useful. For instance, finding the right color space to use can be very important. Max pooling is also used a lot in image recognition networks.

The conclusion is simple: Many deep learning neural networks contain hard-coded data processing, feature extraction, and feature engineering. They may require less of these than other machine learning algorithms, but they still require some.

I am not the only one stating the above; see this article, for instance.

Note: The need for data pre-processing for deep learning is getting more attention. Google just announced tf.Transform to address that need for TensorFlow users.

Deep learning Feature engineering Engineering neural network

Published at DZone with permission of Jean Francois Puget, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Predicting Diabetes Types: A Deep Learning Approach
  • Integrating Data Engineering Into Artificial Intelligence
  • Neural Networks: From Perceptrons to Deep Learning
  • The One-Pixel Threat: How Minuscule Changes Can Fool Deep Learning Systems

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!