Tutorial: Scaling Meteor with MongoDB Oplog Tailing
by
Join the DZone community and get the full member experience.
Join For FreeEver since Meteor 0.7.0 first introduced oplog tailing, we’ve had a lot of users asking us about using the MongoDB oplog with their Meteor applications. As a result, we thought a step-by-step tutorial would help folks get started.
Meteor Oplog Tailing Overview
If you’re still feeling your way around the Meteor framework, you may not know about oplog tailing just yet. The Meteor team released an improvement to observeChanges, which monitors MongoDB’s oplog to avoid extra operations on the database. This improvement significantly reduces the number of queries needed to obtain the freshest changes to your MongoDB.
The “local” database holds the MongoDB oplog
MongoDB’s operations log keeps a rolling record of all inserts, updates and deletes. This log is stored in a special database called “local” which exists on each member of a replica set and does not replicate.
From MongoDB’s documentation on the oplog: “MongoDB applies database operations on the primary and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog.”
The Meteor framework smartly uses the oplog to keep track of changes to your data. This minimizes the number of queries necessary when searching for changes to your data.
Meteor Oplog Tailing Tutorial
We’ve hacked up our own example using Meteor’s oplog tailing with a database hosted on MongoLab (don’t worry, you can still use this tutorial even if you’re not using MongoLab). Here’s a link to this example project’s repo on GitHub.
To give you an idea of what we’re trying to accomplish, we’ll be setting up a simple Meteor app that displays a list of players and their scores. The app will also have an input form for inserting new players and scores to show real-time updates in the client view.
When our project is complete, we’ll observe real-time metrics from two apps, one with oplog tailing configured and one without, which are otherwise identical. The output should look like the following:
Meteor serverFacts output. Oplog tailing enabled output on left, no tailing enabled on right.
Pre-requisites
In this tutorial we’ll assume that you have:
- a MongoDB database (with access to the “local” database)
- your own MongoDB deployment OR
- any for-pay subscription with MongoLab (starts at $15/mo)
- Node.js installed
- Meteor installed
Set up the project
First, create a Meteor project.
> meteor create app
Then navigate into your app to start modifying the project files.
> cd app
Now we’ll remove the “autopublish” package which is not recommended for production use.
> meteor remove autopublish
We’ll also want to run the “facts” package, which contains real-time information about our Meteor server. This will help us determine if oplog-tailing is working.
> meteor add facts
Configure the view
First we’ll define a view to display relevant content to the client. We’ll edit our app.html file to look like the following:
<head>
<title>app</title>
</head>
<body>
<div>
<!-- Players template to display all players in database -->
{{> players}}
</div>
<div>
<!-- User input form to add players -->
{{> form}}
</div>
<div>
<!-- Real-time metrics to confirm oplog tailing -->
{{> serverFacts}}
</div>
</body>
<template name="players">
<h1>List of players</h1>
<ul>
{{#each scorers}}
<li>
<div>
Name: {{ this.name }}
<div>
<div>
Score: {{ this.score }}
</div>
</li>
{{/each}}
</ul>
</template>
<template name ="form">
<form id="myform">
Name: <input type="text" name="name"/>
Score: <input type="number" name="score"/>
<input type="submit" id="submit"/>
</form>
</template>
You’ll notice that the file contains conventional HTML code and Meteor’s templating language, Spacebars (inspired by Handlebars). In the body, we’ll lay out what we want the user to see: a list of players with their scores, a form to add new players, and “serverFacts” metrics.
The “serverFacts” template is a report auto-generated by Meteor that contains real-time information for the current server. This is how we will later verify that our application has oplog-tailing enabled.
Now that we have the templates in place, we need to configure them. We’ll configure the “players” template to iterate over all the players and list their names and scores. The “form” template is straightforward as well – configure it as you would a HTML form.
Create the model
Now we’ll create a new file, models.js, in our project directory to specify a collection to store our documents.
Players = new Meteor.Collection("players");
Our Players model maps to a “players” collection in your MongoDB. You don’t need to create this collection ahead of time as MongoDB will lazily create it for you (it doesn’t already exist) once you insert a document.
Link together the client, server, and MongoDB
In order for the client (browser) view to display all the players, we need to query for them on our database and pass the cursor to the client. We’ll replace the default app.js file to look like the following:
if (Meteor.isServer) {
Meteor.publish("playerData", function() {
return Players.find();
});
// Set which users can see server metrics
Facts.setUserIdFilter(function () {
return true;
});
}
if (Meteor.isClient) {
Meteor.subscribe("playerData");
// Pass query results to the "players" template
Template.players.scorers = function () {
return Players.find();
};
// Grab user input fields
Template.form.events({
'submit form' : function (event, template) {
var name = template.find("input[name=name]").value;
var score = template.find("input[name=score]").value;
// Do input validation
// Insert into database
var data = { name: name, score: score };
Players.insert(data, function(err) {
if(err) throw err;
});
}
});
}
Starting from the top, we have our server side code. This uses Meteor’s publish method to link together the client and server; we’ll cover the intricacies of the publish method in the next section. We also set up the code needed to publish server metrics to the “serverFacts” template that we created back in our app.html file.
We then have our client side code. Similar to the publish method in the server side code, here we use Meteor’s subscribe method, which we’ll also cover in the next section.
Next we create a template helper function that queries on the Players model and passes a cursor to the client (browser) view. This allows the “players” template from our view (app.html file) to iterate through all the Players that are returned and display them in the view.
Finally we create a handler for the “form” template that activates when the form is submitted. Typically you want to put your data validation code here as well, but our example inserts the data directly into the database.
Explore Meteor’s publish and subscribe methods
In order to fully understand what Meteor’s publish and subscribe methods do, it’s important to note that unlike other applications (Rails, Django, etc.) Meteor applications live on both the server and the client. This architecture allows Meteor to send raw data to the client (data on the wire) and access that data instantaneously without having to wait for a round-trip to the server. Meteor ensures that the client and server data (that you specify) are in sync.
In the server code we copied above, we use Meteor’s publish method to help implement oplog tailing. From the Meteor wiki: “Whenever a cursor is returned from the publish function, Meteor calls observeChanges on the cursor and provides it with callbacks which publish the query’s changes to clients.” This means that Meteor continues to watch the published query and calls a callback function when results change.
Previously, Meteor’s only strategy for implementing observeChanges was to re-run the query frequently and calculate the difference between each set of results. With the introduction of the OplogObserveDriver class, Meteor can now read changes from the oplog.
Back to our example, we publish a record set with the name “playerData”. Once you’ve published the query to the “playerData” record set, you need to listen, or subscribe, to that record set on the client side. The subscribe method in our client code tells the server to send this particular set of records to the client, which is stored in the client-side database called MiniMongo.
Now when the data in our MongoDB changes for our players query, our view should reflect those changes in real-time and with minimal cost on the database.
Set up the app
We’re almost there! Next, you’ll need to bundle then extract your project to set up the application.
> meteor bundle app.tgz
> tar -zxvf app.tgz
Create a “local” database user
You’ll need to create a user for your local database so that your Meteor app can access the oplog. You will need to give the Meteor app your user credentials when you run the application. We recommend creating a user with read-only access.
If you’re using MongoLab, you can visit our docs to find instructions on creating a “local” database user.
Once you’re done, be sure to copy the MongoDB URI for your “local” database for the next step.
Run the app
Once you’re set with credentials, you can run your application with the following command:
> PORT=3000 MONGO_URL=<your_uri> MONGO_OPLOG_URL=<your_local_uri> node bundle/main.js
The MONGO_URL points to the MongoDB database that your application reads and writes to, whereas the MONGO_OPLOG_URL should point to your “local” database (which contains the oplog).
See the difference in real-time!
Once your application is running, we can verify if oplog tailing is working. Again, we highly recommend reading the Meteor wiki on the Oplog Observe Driver so you understand the underlying details.
To check if oplog tailing is enabled, you’ll want to verify that the observe-drivers-oplog metric is rendered and the observe-drivers-polling metric is at 0 or not rendered at all. This difference is subtle, but very important!
For extra fun, clone your app and run multiple copies at once to see real-time updates between concurrent clients. I recommend running one copy of your app with oplog tailing and one without. To run the app without oplog tailing, simply leave out the MONGO_OPLOG_URL option in the command.
> PORT=3000 MONGO_URL=mongolab_uri node bundle/main.js
Get the most out of your MongoDB on Meteor
In addition to this tutorial, we highly recommend watching David Glasser’s Devshop 10 talk on oplog tailing. He clearly articulates the background story, problem and solution and provides excellent visual examples.
We hope this tutorial helps you leverage all the tools at your disposal so that you can get the most out of your MongoDB on Meteor. We’re excited to see what you hack up!
Published at DZone with permission of Chris Chang, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Trending
-
Building A Log Analytics Solution 10 Times More Cost-Effective Than Elasticsearch
-
Playwright JavaScript Tutorial: A Complete Guide
-
How To Manage Vulnerabilities in Modern Cloud-Native Applications
-
How Web3 Is Driving Social and Financial Empowerment
Comments