DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • How to Port CV/ML Models to NPU for Faster Face Recognition
  • Unsupervised Learning Methods for Analyzing Encrypted Network Traffic
  • Understanding Neural Networks
  • Understanding the Basics of Neural Networks and Deep Learning

Trending

  • Docker Base Images Demystified: A Practical Guide
  • How Large Tech Companies Architect Resilient Systems for Millions of Users
  • Top Book Picks for Site Reliability Engineers
  • Event-Driven Architectures: Designing Scalable and Resilient Cloud Solutions
  1. DZone
  2. Data Engineering
  3. AI/ML
  4. Visualizing What a Neural Network Thinks

Visualizing What a Neural Network Thinks

We will see how to visualize saliency maps and get an idea of what a neural network considers important in an image.

By 
Luca Venturi user avatar
Luca Venturi
·
Jan. 16, 22 · Presentation
Likes (3)
Comment
Save
Tweet
Share
5.4K Views

Join the DZone community and get the full member experience.

Join For Free

Ok, Neural Networks don’t really "think", but they surely have an opinion. If you feed a Convolutional Neural Network, let’s say, a classifier with an image, it will tell you what it "thinks" is there, but sometimes you might wonder what contributed to that particular decision, as an attempt to try to "debug" your network.

For example, when designing a gender classification Neural Network, I noticed that the network was paying attention to the ears, and it learned to recognize them. It turned out ears can be used to detect gender, so it was a good sign that my network was looking for them.

In the rest of the article, we will see how to use saliency maps to get an idea of what a Neural Network considers important. The following content is mostly an extract from the book Hands-On Vision and Behavior for Self-Driving Cars that I wrote for Packt Publishing, with the help of Krishtof Korda; in the previous part of the chapter, we trained a Neural Network to learn how to self-drive a car in the simulator Carla, and the model of the network is stored in a variable called model.

Visualizing the Saliency Maps

To understand what the neural network is focusing its attention on, we should use a practical example, so let’s choose an image:

Test Image For Saliency MappingTest image

If we had to drive on this road, as humans, we would pay attention to the lanes and the wall, though admittedly, the wall is not as important as the last lane is before that.

We already know how to get an idea of what a CNN (short for Convolutional Neural Network) such as DAVE-2 is taking into consideration: as the output of a convolution layer is an image, we can visualize it as follows:

Part of The Activations of The First Convolutional LayerPart of the activations of the first convolutional layer

This is a good starting point, but we would like something more. We would like to understand which pixels contribute the most to the prediction. For that, we need to get a saliency map.

Keras does not directly support them, but we can use keras-vis. You can install it with pip, as follows:

Shell
sudo pip install keras-vis


The first step to getting a saliency map is to create a model that starts with the input of our model but ends with the layer that we want to analyze. The resulting code is very similar to what we saw for the activations, except that for convenience, we also need the index of the layer:

Python
conv_layer, idx_layer = next((layer.output, idx) for idx, layer in enumerate(model.layers) if
                                 layer.output.name.startswith(conv_name))

act_model = models.Model(inputs=model.input, outputs=[conv_layer])


While not necessary in our case, you might want to change the activation to become linear, then reload the model:

Python
conv_layer.activation = activations.linear
sal_model = utils.apply_modifications(act_model)


Now, it is just a matter of calling visualize_saliency():

Python
grads = visualize_saliency(sal_model, idx_layer,
filter_indices=None, seed_input=img)
plt.imshow(grads, alpha=.6)


We are interested in the saliency map of the last layer, the output, but as an exercise, we will go through all the convolutional layers to see what they understand.

First Test Visualization

Let’s see the saliency map for the first convolutional layer:

Saliency Map of The First Convolutional LayerSaliency map of the first convolutional layer

Not very impressive, as there is no saliency and we only see the original image.
Let’s see how the map of the second layer looks like:

Saliency Map of The Second Convolutional LayerSaliency map of the second convolutional layer

This is an improvement, but even if we see some attention in the middle line, on the wall, and the land after the right lane, it is not very clear. Let’s see the third layer:

Saliency Map of The Third Convolutional LayerSaliency map of the third convolutional layer

Now we are talking! We can see great attention on the central and left line, and some attention focused on the wall and the right line. The network seems to be trying to understand where the road ends. Let’s also see the fourth layer:

Saliency Map of The Fourth Convolutional LayerSaliency map of the fourth convolutional layer

Here, we can see that the attention is mostly focused on the central line, but there are also sparks of attention on the left line and on the wall, as well as a bit on the whole road.

We can also check the fifth and last convolutional layer:

Saliency Map of The Fifth Convolutional LayerSaliency map of the fifth convolutional layer

The fifth layer is similar to the fourth layer, plus with some more attention on the left line and on the wall.

We can also visualize the saliency map for dense layers. Let’s see the result for the last layer, which is what we consider the real saliency map for this image:

Saliency Map of The Output LayerSaliency map of the output layer

The last saliency map, the most important one, shows great attention to the central line and the right line, plus some attention on the upper-right corner, which could be an attempt to estimate the distance from the right lane. We can also see some attention on the wall and the left lane. So, all in all, it seems promising.

Second Test Visualization

Let’s try with another image:

Second Test ImageSecond test image

This is an interesting image, as it is taken from a part of the road where the network has not been trained, but it still behaved very well.
Let’s see the saliency map of the third convolutional layer:

Saliency Map of The Third Convolutional LayerSaliency map of the third convolutional layer

The neural network seems very concerned with the end of the road and it seems to have detected a couple of trees as well. If it was trained for braking, I bet it would do so!

Saliency Map of The Output LayerSaliency map of the output layer

This is pretty similar to the previous one, but there is some attention to the central line and the right line, and a tiny amount on the road in general. Looks good to me.


Let’s try with the last image, taken from the training to teach when to turn right:

Third Test ImageThird test image

This is the final saliency map for it:

Saliency Map of The Output LayerSaliency map of the output layer

You can see that the neural network is giving attention mostly to the right line, also keeping an eye on the whole road and with some spark of attention dedicated to the left line.

As you can see, the saliency map can be a valid tool to understand the behavior of the network a bit more and do a kind of sanity check on its interpretation of the world.

Last Words

This concludes the article. In the book, the rest of the chapter explains how to integrate the Neural Network with the simulator Carla, so you can actually experience a self-driving car, though in a simulator; it also explains how to use generators to enable training with bigger datasets and to do more advanced data augmentation.

If you liked this article and if you are curious about some of the technologies used by self-driving cars, please consider the Hands-On Vision and Behavior for Self-Driving Cars book. It talks also about Lane Detection, Neural Networks, Object Detection, Semantic Segmentation, sensors, lidars, maps, control, and more.

neural network Network

Published at DZone with permission of Luca Venturi. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • How to Port CV/ML Models to NPU for Faster Face Recognition
  • Unsupervised Learning Methods for Analyzing Encrypted Network Traffic
  • Understanding Neural Networks
  • Understanding the Basics of Neural Networks and Deep Learning

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!