Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Using Firebase Cloud Functions to Manage a Compound Key Index

DZone's Guide to

Using Firebase Cloud Functions to Manage a Compound Key Index

Firebase Cloud Functions are a nice solution to the problem of using composite key indexes and can be used in the same way as a stored procedure in a traditional RDBMS.

· Cloud Zone
Free Resource

What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDB. Sign up and you'll receive instructions for how to get started!

In Firebase, Google’s ‘NoSQL’ database, you can declare indexing by any of a node’s children in order to optimize retrieval by that field. For instance, under a node called ‘artists’ where the keys are the artist’s ‘uid’ and the values are artist objects, the default indexing is performed by uid.

The /artists node in our Firebase DB

While you could retrieve the artists, specifying in the query that they be ordered by display name, it’s not efficient. If you expect to retrieve artists by display name often, then you should declare an index, like so:

"artists": {
   ".indexOn": ["user/name/display"],
     ...
}

But let’s say you want to support a predictive search function where the user chooses a genre and begins typing an artist’s name.

As they type, users should see a drop-down list containing the first five artists in the chosen genre whose name matches the input so far. You can specify a limit to the number of items returned in the query, so as long as they’re in the desired order, populating the predictions list should be simple and quick.

Firebase Supports Simple Key Indexing

You’ll notice in the example above that the .indexOn field takes an array — you can index on more than one field, like genre name. However, the results wouldn’t ALSO be in order by artist display name. It would be optimized for retrieval by artist display name OR by genre, but not by both. For that, you need compound keys, which aren’t (currently) supported in Firebase’s rule language. That is, in the .indexOn field, we cannot specify multiple fields to make up a single compound key. I suspect that will happen at some point, but for now, we have to tackle the problem ourselves.

We could create an ‘artist_by_genre’ field on the artist object and tell Firebase to index on that field. We’d need to concatenate the genre and artist display name together on the client and store it in that property before saving the artist object.

If the fields making up the compound key aren’t on objects inside collections, then this approach is fine. For instance, postal code and artist name would work and we could retrieve artists alphabetically within a given postal code as easily as by any other simple field.

In our case, however, we need to support multiple index entries for each artist, since an artist (as in the one shown above), can work in multiple genres.

Implementing a Compound Key Index

The 'artists by genre' index

First, we need to create another node to hold the key/value pairs that will make up our index. The keys will be genre name and artist display name concatenated.

As you can see, we are easily able to accommodate multiple index entries for the same artist using this key scheme.

And since the index is naturally ordered by the key, the data will be alpha by genre, and within that, alpha by artist display name.

From the data structure alone, you can see that using a compound key strategy is not that hard to implement. The only question is what should the key’s value be?

We could just make it be the uid of the artist, and then make a separate call for the artist node, but that’d be chattier than is desirable in a predictive search query, where we’re already making queries to get the first five artists in the genre whose display names match the user’s input as they type. Making another five calls when each of those queries return would be way too much.

You shouldn’t fear duplicating data — disk is cheap and speed is dear. The important thing is to store all the data you need to fulfill every use case that reads this index.

So we could just write the entire artist object here at this node. But it might be a big object, and contain much that is extraneous for our purposes. In our case, we just want to have the drop-down list contain the name of the artist, and possibly a thumbnail profile image. When the user actually selects an artist from the list, then we can make a single call to grab the full artist profile object from the /artists node.

Maintaining the Index

When an artist writes a profile object to the /artists node, the artists_by_genre index also needs to be updated. The first time the profile is written, it is sufficient to just write a key/value pair for each genre in the artist’s profile. But when they update their profile, changing their name or perhaps modifying the genre list, it gets more involved. You need to remove the old entries and add the new ones. That could mean four or five writes for each artist when updating their profiles.

Now, in this particular scenario, doing five writes to update an artist by genre index might not be that much chattiness even at scale. You could optimize that by comparing the old genre list and the new one and determining if the name had changed, of course.

But you might have a scenario where you are writing and deleting entries in a compound key index more frequently and with higher volume. Not only is it cumbersome to include the client code to manage this index, it’s also much chattier than one might like.

Using Firebase Cloud Functions as Stored Procedures

An exciting new feature of Firebase is Cloud Functions. A Firebase Cloud Function is just that: a function, written in JavaScript, deployed in Node.js, which can be triggered by a variety of inputs:

It’s the ultimate in micro-services. Inside your Cloud Function, you can do pretty much anything your heart desires. In our case, we’re going to use it like a traditional SQL Stored Procedure.

When a child of /artists is written, we want to have our Cloud Function fire and update our artist_by_genre index for us.

Having a database up and running already, we’ve initialized our project for Firebase and are logged in through the Firebase CLI. In order to add a Cloud Function, we just need to enable that feature. In our project’s folder, we’ll initialize Firebase Cloud Functions:

firebase init functions

Now the project has the following Firebase-related configurations (the functions folder was just added):

project
 +- .firebaserc    # EXISTING - Hidden file that helps you quickly switch between
 |                 #            projects with `firebase use`
 |
 +- firebase.json  # EXISTING - Describes properties for your project
 |
 +- functions/     # NEW - Folder containing all your functions code
      |
      +- package.json  # NEW - The npm package file describing your Cloud Functions
      |
      +- index.js      # NEW - Main source file for your Cloud Functions code
      |
      +- node_modules/ # NEW - Folder for Cloud Functions dependencies

The functions/index.js file is where we will add our Cloud Function(s).

exports.indexArtist = functions.database.ref('/artists/{uid}')
    .onWrite(event => {

        // All the write promises this function will create
        let writes = [];
        let artistByGenreIndex = admin.database().ref('/indexes/artist_by_genre');

        console.log('Triggered by artist', event.params.uid, event.data.val().user.name.display);

        // Remove any previous entries since genre or display name may have changed
        var previous = event.data.previous.exists() ? event.data.previous.val() : null;
        let deletes = [], key, val;
        if (previous && previous.genres){
            console.log('Removing old index entries...');
            previous.genres.forEach( genre => {
                key = genre.name + previous.user.name.display;
                deletes.push( artistByGenreIndex.child(key).set(null) );
            });
            console.log(deletes.length+' entries removed.');
        }

        // If any previous entries were deleted, wait until they finish
        Promise.all(deletes).then(addKeys).catch(addKeys);

        // You must return a Promise when performing async tasks in a Firebase Cloud Function
        return Promise.all(writes);

        // Add the keys for all Genres this Artist has chosen
        function addKeys() {
            const current = event.data.exists() ? event.data.val() : null;
            if (current && current.genres) {
                console.log('Indexing artist: ', event.params.uid, current.user.name.display);
                current.genres.forEach( genre => {
                    key = genre.name + current.user.name.display;
                    val = {key: key, artist: {user: current.user}, genre:genre};
                    writes.push( artistByGenreIndex.child(key).set(val) );
                });
            }
        }
    });

Notice in the above code, we did not have to specify any credentials. That's because it's deployed to the server where the database lives and is part of our project. Also, we are using the Firebase Admin SDK to write the index, which means we can deny write privileges to everyone in the database ruleset for the artist_by_genre node. Here's a GitHub Gist of that code.

Artist by genre index rules

Now, with our function written and rules applied, all we need to do is deploy the function and then write to the database to see if it worked.

firebase deploy --only functions

Deploy Firebase Cloud Functions

At any time, on the Firebase Console for our project, we can see what functions are deployed, and some info about their executions. And after writing an artist to the database, we can immediately see the Cloud Function’s output by choosing that function and selecting ‘View logs.’

Deployed Cloud Functions

Firebase Cloud Function Log

All of our console output is available in addition to timing and system messages, such as the warning that we need to upgrade to a paid plan if we want to access external APIs or raise our usage quotas.

And the proof is in the pudding, so to speak. The artist_by_genre index contains the proper data (the above index entries were written by this Cloud Function).

Conclusion

Composite key indexes with multiple keys per source node are often needed but currently unsupported by Firebase. While we can create a special node to hold the key/value pairs that represent such an index, it could be cumbersome and chatty to include in a client application. And if multiple client applications with different codebases are accessing the same database, then duplicating the logic to maintain the composite key indexes could be even more onerous.

Firebase Cloud Functions are a powerful solution to the problem and can be used in the same way that you would use a stored procedure in a traditional RDBMS. They are easy to build, deploy, and maintain. This centralizes the mundane indexing logic and reduces network chatter.

I hope this helps you out in your next big Firebase project.

What if you could learn how to use MongoDB directly from the experts, on your schedule, for free? We've put together the ultimate guide for learning MongoDBSign up and you'll receive instructions for how to get started!

Topics:
firebase ,cloud functions ,database indexes ,stored procedures ,cloud

Published at DZone with permission of Cliff Hall. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}