Tutorial: How to Build a Progressive Web App (PWA) with Face Recognition and Speech Recognition
In this tutorial, I will discuss advanced PWA features that provide access to your hardware APIs. We are going to build an app with Face Recognition and Speech Recognition. This is now possible on the Web/Browser! The native experience users see on native apps is now brought to the web! This would open up a whole new approach.
Join the DZone community and get the full member experience.Join For Free
This is a follow up to the second tutorial on PWA. You can also follow this tutorial if you haven't followed the second one or my first tutorial about PWA. We are going to focus on some new Web APIs, such as:
- Face detection API, for face recognition in the browser. https://justadudewhohacks.github.io/face-api.js/docs/index.html
- Web speech API, for enabling "Speech to text" in this app. https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API
We add these APIs to our existing PWA for taking "selfies". With face detection we predict your emotion, your gender and your age.
To record an accompanying text ("Speech to text") for your "selfie", you can easily use the Web Speech API.
Experimental Web Platform features
The above APIs only work if you have enabled "Experimental Web Platform features" in your Chrome browser via the url: chrome: // flags
As a starting point for the tutorial, clone the following Github repository:
Then in your terminal move to the following directory:
and install the dependencies through:
Open your app on: http:// localhost:8080
Public url for your mobile
There are many ways to access our localhost:8080 from a remote mobile device. You can use ngrok for this (https://ngrok.com/).
Install ngrok via:
And run the following command in your terminal:
This command generates a public url for you. Then browse to the generated url on your mobile in Chrome.
Face detection / face recognition is one of the most used applications of Artificial Intelligence. The use of face recognition has increased in recent years.
In this tutorial we will expand the existing app with face recognition that even works in the browser. We predict your emotion, your gender and your age based on your "selfie". We are using the Face-api.js here.
The output of this app looks as shown below:
The steps to implement the above are as follows:
Step 1 face-api
Step 2 models
Models are the trained data that we will use to detect features of your "selfie". These models are already available in the folder: public/src/models
Step 3 index.html
In the index.html file we import:
- the existing facedetection.css for the styling (see listing 1);
- face-api.min.js, this is the Face detection API for processing the model data and extracting the features (see listing 3);
- facedetection.js where we will write our logic.
First, import the styling into the index.html
Place the code below in the index.html file, directly below the tag: <div id="create-post">
We use the existing video tag to be able to take a "selfie"(see listing 2). And a "result container" for predicting your emotion, your gender and your age.
Place the code below at the bottom of the index.html, so that we can use the Face-detection API:
Step 4 import models into the PWA
Here we create a separate function in feed.js to start the video streaming (listing 4).
Move the following existing code from the initializeMedia()-function into a separate function called startVideo() (see result code: listing 4)
This function is responsible for the video streaming:
In the existing feed.js we use Promise.all to load the models for the face API asynchronously. Once these models are properly loaded, we call the created startVideo() function (see listing 5).
Place the following code at the bottom of the initializeMedia() function:
Step 5 Implement the facedetection.js
Below are the functions of the Face detection API, which we use in our app:
faceapi.detectSingleFace uses the SSD Mobilenet V1 Face Detector. You can specify the face recognition by passing the videoPlayer object an options object. To detect multiple faces, replace the detectSingleFace feature with detectAllFaces.
This function is used to detect 68 face landmarks.
This function detects all faces in an image and recognizes face expressions from each face and returns an array.
This function detects all faces in an image, estimates the age and gender of each face and returns an array.
Place the code below in the existing file with the name: facedetection.js, under the existing code (see listing 6).
The above functions are called in here to perform face recognition.
First, a playing event handler is added to the videoPlayer, which responds when the video camera is active.
The variable videoPlayer contains the HTML element <video>. Your video tracks will be rendered in this.
Then a canvasElement is created under the name canvasForFaceDetection. This is used for face recognition. This canvasForFaceDetection is placed in the container faceDetection.
The setInterval() function repeats the faceapi.detectSingleFace function at a time interval of 100 milliseconds. This function is called asynchronously using async / await and finally the results of the face recognition are displayed in the fields: emotion, gender and age.
Web Speech API
The interface we are going to build for the Web Speech API will look like the one shown below (see figure 4). As you can see, the screen contains an input field to enter text. But it is also possible to record text using "Speech to text". For this you have to click on the microphone icon in the input field.
Under this input field you can also choose a desired language via a select box.
The steps to implement the above are as follows:
Step 1 index.html
In the index.html file we import:
- the existing speech.css for the styling (see listing 7);
- speech.js where we will write our logic for the "Speech to Text" feature (see listing 9).
First, import the styling into index.html
Place the code below in the index.html, directly below the tag: <form> (see listing 8)
In the <div id ="info"> - section the info texts are placed that can be displayed as soon as this API is used.
With the startButton onclick-event, you can start and use the API.
Finally, with updateCountry onchange-event you can select a desired language via a select box.
Place the code below at the bottom of index.html, so we can use the Web Speech API (see listing 9):
Step 2 Implement the Web Speech API
This code in the speech.js initializes the Web Speech Recognition API (see listing 10):
If the "webkitSpeechRecognition" is available in the window object, then a "webkitSpeechRecognition" object is created via: recognition = new webkitSpeechRecognition ();
We then set the following properties of the Speech API:
recognition.continuous = true
This property determines whether results are continuously returned for each recognition.
recognition.interimResults = true
This property determines whether the Speech recognition returns intermediate results.
This event handler is executed when the SpeechRecognition API is started (see Listing 10).
The following info text is displayed: "Speak now". Then an animation of a pulsating microphone is shown: mic-animate.gif (see image)
This event handler is executed when the SpeechRecognition API returns a result.
The SpeechRecognitionEvent "results property" returns a 2-dimensional SpeechRecognitionResultList. The isFinal property determines in a "loop" whether a result text from the list is "final" or "interim". The results are also converted to a string with the transcript property.
This event handler is executed when Speech Recognition is terminated.
No info text is displayed, only the default icon of a microphone.
This event handler handles the errors. A matching info text is also shown with the error message.
Starting the speech recognition
Add the following code at the top of the already existing speech.js, to start the Web Speech API, using the startButton event (see listing 11):
This code starts the Web Speech Recognition API using recognition.start(). This start() function triggers a start event and is handled in the event handler recognition.onstart() (see Listing 10).
Furthermore, the selected language is set with recognition.lang. And finally, a pulsating microphone image is shown.
The Web is getting more sophisticated day-by-day. More native features are being brought on board, this is due to the fact that the number of web users is far greater than native app users. The native experience users see on native apps is brought to the web to retain them there without the need to go back to the native apps.
After this introduction, you can continue with an extensive tutorial that you can find in: https://github.com/petereijgermans11/progressive-web-app/tree/master/pwa-workshop.
See also my previous article. In this article I will discuss some advanced PWA features that provide access to your hardware APIs, like: Media Capture API, Geolocation API and the Background Sync API.
And take a look into my first article. In this article I discuss some basics behind building PWAs and then provide a comprehensive tutorial on creating your first PWA.
Follow or like me on twitter https://twitter.com/EijgermansPeter
Opinions expressed by DZone contributors are their own.