Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

koa-pageable: Spring-Style Pagination in JavaScript

DZone's Guide to

koa-pageable: Spring-Style Pagination in JavaScript

A developer discusses a recent project he worked on in JavaScript and Node.js, along with a Spring-based middleware, to build a RESTful service.

· Integration Zone ·
Free Resource
CRM integration has become the cornerstone to meeting initiatives across organizations. Explore the top 6 value-driven Salesforce CRM integrations ebook.  

A JVM Developer’s Journey Into the Modern JS World

After working in the JVM/Spring ecosystem for over a decade, I recently worked on a JavaScript/Node.js project building a RESTful service application using koa.

Here are a few of my high-level takeaways:

  • Wow, JavaScript has improved significantly since I last used it a decade ago.
  • Flow types really go a long way to improving code correctness and refactorability.
  • Huge YES to async/await.
  • The npm/yarn, babel, webpack stack make Maven look pretty good...
  • I really liked the middleware approach. They’re like much more usable servlet filters.
  • The overall Node ecosystem appears to be really lacking in well-defined patterns for standard “Enterprise” (a.k.a. large multi-developer application) system Quality Attributes (non-functionals).
  • Just a couple examples of “enterprise” level features I found lacking:
    • Logging — There appears to be no straightforward way to output which file a log statement originated in (other than hardcoding). No standardized packaging means no ability to set log level by package. No MDC.
    • Error Handling — Many libraries don’t even bother declaring their own error types, they just throw Error instances even for well-defined cases. This means that error handling often involves parsing out message strings. 
    • Documentation — There is no good solution for generating documentation from JSDoc annotated source code. The most common (jsdoc, esdoc, documentation.js) all have limits and support different subsets of the language and features (e.g. lacking basic ES6 support, improper/non-configurable class inheritance handling). It surprised me that there is no frequently updated project in this space.

On the whole, though, I really enjoyed the experience. Personally, I’m still more comfortable with the many benefits provided by a statically typed language (say, Kotlin) along with a framework that provides most “table-stakes” functionality out of the box (e.g. Spring Boot). That said, I certainly see value in the Node ecosystem and, given the right use case, would be glad to develop on it in the future.

I Couldn’t Find it, So I Built it

One Spring nicety that I immediately missed was Spring Data's Pagination support. I looked at a number of the available libraries but while they all provided the base pagination functionality, they did not support all of the use cases I was used to seeing from my pagination library.

Desired features:

  • Support offset (page number * page size = offset) based pagination.
  • Use query parameters (not headers).
  • Allow the client to request the response to be Sorted.
  • Provide a consistent response format for all paginated requests including the actual returned data and pagination metadata such as current page number, current page size, the total number of pages, total number of elements, etc.
  • Provide flow definitions for strong typing.

So, as I was unable to find a library that met the requirements, I decided to build one leveraging a number of patterns that I’d found very useful in Spring.

While building out the library, we identified some additional desirable features. One of the consumers of our API is a React Native application using redux for state management. The front end team identified that it would also be useful to allow the API client to control the format of the data. While the standard format is, of course, an ordered list, front-end applications using a state management system like redux often normalize the data coming in, storing entities in a map-like data structure. We don’t necessarily want to solve that problem or become overly opinionated, but we can provide a quick win for redux-like consumers of an API if there is the ability to return data from the API represented as a map instead of an array.

koa-pageable

koa-pageable is a middleware for pagination in koa inspired by Spring Data's Pagination support.

It provides control to the clients of your API, allowing them to specify the amount, order, and format of the data they can retrieve.

If you’re familiar with Spring Data then many of the same domain terms will be familiar to you: PageablePageSort. It goes a step further than Spring Data in that it also allows the client to partially specify the structure of the returned data.

Let’s pretend that you have a RESTful endpoint /people that is backed by a datastore containing 1000 people. If a client wanted to retrieve a list of people in batches of 10, sorted by lastname, they could issue the below request to get the second such page (pages are 0-indexed): 
GET /people?page=1&size=10&sort=lastname

Usage

Note: examples below include flow types as I’m a big fan of static typing. These are definitely optional though.

First, install the library via your favorite dependency manager, I like yarn.

yarn add @panderalabs/koa-pageable

Then add the middleware function, and pass the pageable instance that it populates into ctx.state.pageable to your data tier:

// @flow
import { Pageable, IndexablePage, paginate } from '@panderalabs/koa-pageable';
import type { Context } from 'koa';
import Koa from 'koa';
import Router from 'koa-router'

const app = new Koa();
const router = new Router();
app
  .use(router.routes())
  .use(router.allowedMethods());

router.get('/people', paginate, async (ctx: Context) => {
  const pageable: Pageable = ctx.state.pageable;
  const result: IndexablePage<number, Person> = service.getData(pageable);
  ctx.body = result;
};

Finally, you must ensure that your data tier properly implements the client’s intention (sorting the request, limiting the data size). The exact mechanism for doing this will depend on your data-access/ORM framework, but this is what it looks like using objection:

// @flow
import { IndexablePage, Pageable, Sort, } from '@panderalabs/koa-pageable';
import type { QueryBuilder } from 'objection';

function getData(pageable: Pageable): IndexablePage<number, Person> {
  const pageNumber = pageable.page;
  const pageSize = pageable.size;
  const sort: Sort  = pageable.sort;

  const queryBuilder: QueryBuilder = Person.query().where('age', '>=', 21).page(pageNumber, pageSize);

  //If there is a sort, add each order element to the query's `orderBy`
  if (sort) {
    sort.forEach((property, direction) => queryBuilder.orderBy(property, direction));
  }  
  const result = await query.execute();

  return new IndexablePage(result.results, result.total, pageable); 
}

There is only one other concept to be aware of, and it relates to the format of the data serialized back to the client.

Array or Index

koa-pageable can serialize the result of a request in two different formats depending on the client’s request (value of the indexed query parameter).

Array Format

Content returned in a simple ordered array.

GET /people?page=1&size=2&sort=firstname,lastname:desc&indexed=false
{            
  "number": 1,
  "size": 2,
  "sort": [
    {
      "direction": "asc",
      "property": "firstname"
    },
    {
      "direction": "lastname",
      "property": "desc"
    }
  ],
  "totalElements": 1000,
  "totalPages": 500,
  "first": false,
  "last": false,
  "indexed": false,
  "content": [
    {
      "id": 202,
      "firstName": "Bob",
      "lastName": "Smith"
    },
    {
      "id": 200,
      "firstName": "Bob",
      "lastName": "Jones"
    }
  ],
  "numberOfElements": 2
}

Indexed format

Returned as an ordered array of ids, and a corresponding index which is a map of {id: content item}. Note: in order to automatically return data in an indexed format, the underlying data type (in this, case the person class) must have a property named id, the values of which must be unique within the dataset.

GET /people?page=1&size=2&sort=firstname,lastname:desc&indexed=true
{
  "number": 1,
  "size": 2,
  "sort": [
    {
      "direction": "asc",
      "property": "firstname"
    },
    {
      "direction": "lastname",
      "property": "desc"
    }
  ],
  "totalElements": 1000,
  "totalPages": 500,
  "first": false,
  "last": false,
  "ids": [
    202,
    200
  ],
  "index": {
    "200": {
      "id": 200,
      "firstName": "Bob",
      "lastName": "Jones"
    },
    "202": {
      "id": 202,
      "firstName": "Bob",
      "lastName": "Smith"
    }
  },
  "numberOfElements": 2
}

That’s it! That’s everything you need to know to bring some Spring-inspired paginated goodness to your koa Node.js app!

For more detailed information, you can check out the project readme on GitHub and the API Docs.

We’ve already used this framework on a number of projects at my employer Pandera Labs, and plan to continue supporting it in the future.

I’d love input, feedback, collaboration on the library and welcome any Issues and/or Pull Requests.

Sync, automate, and notify lead to customer changes across marketing, CRM, and messaging apps in real-time with the Cloud Elements eventing framework. Learn more.

Topics:
spring framework ,node.js ,javascript ,integration ,restful

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}