How to Improve the Performance of ESLint in Gulp

As you may already know, GulpJS is a JavaScript task runner that lets you automate several tasks during development and ESLint checks the code quality based on defined rules. A quite popular library which helps to combines these two tools is gulp-eslint. 

The default configuration to make linting work with help of gulp is quite trivial:

const {src, task} = require('gulp');
const eslint = require('gulp-eslint');

task('default', () => {
    return src(['scripts/*.js'])
        // eslint() attaches the lint output to the "eslint" property
        // of the file object so it can be used by other modules.
        .pipe(eslint())
        // eslint.format() outputs the lint results to the console.
        // Alternatively use eslint.formatEach() (see Docs).
        .pipe(eslint.format())
        // To have the process exit with an error code (1) on
        // lint error, return the stream and pipe to failAfterError last.
        .pipe(eslint.failAfterError());
});

In the beginning of a project, it works really well with a reasonable performance. The problems appear when the code base becomes massive and to check the code costs not seconds anymore but minutes. It becomes really annoying if you want to have a fast feedback and don't want to switch to another activity during this waiting time or especially when you are under time pressure to deliver the story as soon as possible. 

ESLint has a good feature to turn on a cache to overcome this problem and verify only the updated files. The issue that the passing this property through a configuration in gulp-eslint doesn't work, because of the way how it is integrated. That branch of code where the caching is happening even not executed. 

One of the solutions to this problem could be to create a small wrapper around it to mimic a caching logic. Let's first see how our gulp can look like:

import cache from 'cache'; // this file we will create

const lintStream = (stream) => stream
    .pipe(cache.cacheAndFilter({filename: '.eslintstore'})) // extra line
    .pipe(eslint())
    .pipe(eslint.format())
    .pipe(eslint.failOnError())
    .pipe(cache.updateCache()); // extra line

gulp.task('lint', () => lintStream(gulp.src(`${paths.webSrcDir}/**/*.js`)));

On line 4, we will collect information about all files and filter out already verified files on next lint run. And line 8 will persist/update the cache of modified and passed the verification by lint files.

First, we need to scan all file paths for the given file stream with respect to the last modified timestamp. To simplify work with streams we can use through2 module.

import through from 'through2';

const files = {};

const cacheAndFilter = () => {
    const collect = (file, enc, cb) => {
        files[file.path] = file.stat.mtime;
        cb(null, file);
    };

    return through.obj(collect);
};

export default {
    cacheAndFilter
}

The logic is only on line 7, where we collect the information about each file. Line 8 says that the received file stream will be passed to the next pipe without any modification. For example, to filter that file stream out will require such implementation

cb(null, null);

Once we collected all information about files, we can start persisting the timestamps of validated files having no linting issues. When ESLint process each file it modifies it and creates another file object contacting additional data. In our case, we care about `file.eslint.errorCount`, If it is not 0 than we don't update timestamp (line 24).

import through from 'through2';
import PluginError from 'plugin-error';
import fs from 'fs';

let cachedFiles = {};
let options;
const files = {};

const cacheAndFilter = (opt) => {
    if (!opt.filename) {
        return cb(new PluginError('gulp-eslint-cache', 'Provide filename for caching'));
    }
    options = opt;

    const collectAndFilter = (file, enc, cb) => {
        files[file.path] = file.stat.mtime;
        cb(null, file);
    };

    return through.obj(collectAndFilter);
};

const updateCache = () => through.obj((file, enc, cb) => {
    if (!file.eslint.errorCount) {
        fs.writeFile(options.filename, JSON.stringify({...cachedFiles, ...files}), 'utf8', (err) =>
            cb(err ? new PluginError('gulp-eslint-cache', err) : null, file));
    } else {
        cb(null, file);
    }
});

export default {
    cacheAndFilter,
    updateCache
}

And the last missing part is to actually apply our caching knowledge and filter the files which we would like to send to ESLint. As this part is the performance bottleneck which we are solving.

The added code between lines 22 and 27 does it by looking up only modified files. It compares the current timestamp of each file with the timestamp saved in .eslintstore in the previous run. If the file is new, it won't be in .eslintstore so then we have to lint it. Or if the timestamp of this file from the previous run is older than actual file timestamp then we have also to lint this file again.

import through from 'through2';
import PluginError from 'plugin-error';
import fs from 'fs';

let cachedFiles = {};
let options;
const files = {};

const cacheAndFilter = (opt) => {
    if (!opt.filename) {
        return cb(new PluginError('gulp-eslint-cache', 'Provide filename for caching'));
    }
    options = opt;

    if (fs.existsSync(options.filename)) {
        cachedFiles = JSON.parse(fs.readFileSync(options.filename, 'utf8'));
    }

    const collectAndFilter = (file, enc, cb) => {
        files[file.path] = file.stat.mtime;

        const currentTimestamp = cachedFiles[file.path];
        if (!currentTimestamp || currentTimestamp && new Date(currentTimestamp).getTime() < file.stat.mtime.getTime()) {
            cb(null, file);
        } else {
            cb(null, null);
        }
    };

    return through.obj(collectAndFilter);
};

const updateCache = () => through.obj((file, enc, cb) => {
    if (!file.eslint.errorCount) {
        fs.writeFile(options.filename, JSON.stringify({...cachedFiles, ...files}), 'utf8', (err) =>
            cb(err ? new PluginError('gulp-eslint-cache', err) : null, file));
    } else {
        cb(null, file);
    }
});

export default {
    cacheAndFilter,
    updateCache
}

I didn't want to create yet another npm package for that. First, because the logic is quite compact and each project has own specifics that this implementation might not fit exactly project's needs. But having this as a base and if necessary tweaking some parts can significantly improve your experience with linting. Especially if you already use gulp-eslint in your project and didn't make any performance improvements so far.