Pagination With Spring Data Elasticsearch 4.4

Some time ago, I wrote the Introduction to Spring Data Elasticsearch 4.1 article. As I promised, I want to continue with a search feature. More specifically, the topic is its pagination part. Therefore, this article has these goals:

1. Update my sat-elk project to use Spring Data Elasticsearch 4.4
2. See several options to paginate results

Note: I recommend reading the previous article in order to understand the City domain which is used below. It's not needed from the technical point of view, but it can help to understand the presented examples more.

In This Article, You Will Learn

• How to configure Spring Data Elasticsearch 4.4 in a project. 
• How to paginate a large response result using Spring Data Elasticsearch. 

Spring Data Elasticsearch Setup

Our goal is to have an application to manage data via the Spring Data Elasticsearch in Elasticsearch. You can find the detailed guide in my previous article Introduction to Spring Data Elasticsearch 4.1. In this article, you can find only the simple steps with the highlighted differences.

First, let's check the changes from the last article.

Changes

The last article used Spring Data Elasticsearch in version 4.1, but the latest version (at the time of writing this article) is version 4.4. You can find all the changes here.

We should keep in mind the compatibility matrix that contains the compatible versions of the main technologies - Spring framework, Spring Boot, Spring Data Release Train, Spring Data Elasticsearch and of course Elasticsearch itself.

Elasticsearch Configuration

The fully detailed setup of an Elasticsearch cluster was described (as it was already mentioned) in my previous article. We use the same steps, but with just minor changes.

Custom Network

docker network create sat-elk-net

Elasticsearch

docker run -d --name sat-elasticsearch --net sat-elk-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.17.4

Note: we use the docker image for Elasticsearch 7.17.4 as defined by the compatibility matrix.

Disable XPack in Elasticsearch

2022-07-18 08:57:05.821  WARN 6196 --- [nio-8080-exec-1] org.elasticsearch.client.RestClient      : request [POST http://localhost:9200/_bulk?timeout=1m] returned 1 warnings: [299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."]

In the DEV environment, we can disable X-Pack security as:

docker exec -it <container_id> bash
cd /usr/share/elasticsearch/config
echo "xpack.security.enabled: false" >> elasticsearch.yml

See: https://stackoverflow.com/questions/67993633/how-to-fix-this-in-error-rails-warning-299-elasticsearch-built-in-security-fea

ElasticHQ

docker run -d --name sat-elastichq --net sat-elk-net -p 5000:5000 elastichq/elasticsearch-hq

Maven Configuration

We use the spring-boot-starter-data-elasticsearch dependency in our Maven project (pom.xml) as shown below. We can find the latest available version in the Maven Central repository.

<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
	<version>2.7.4</version>
</dependency>

Pagination

The deprecated search method  (as mentioned in the previous article) is removed in the latest Spring Data Elasticsearch. Therefore, we don't have a straightforward way to use pagination except for the static query.

Note: the official documentation is not up-to-date, because it contains the Filter Builder chapter with the usage of searchForPagemethod. However, this method is not available anymore.

The only possible solution for building a custom query with the pagination feature is to use the search method on the ElasticsearchOperations instance. This instance is auto-configured by Spring Data Elasticsearch.

Let's look at the ElasticsearchOperations usage.

SearchHits Response

A searchHits method (see line 7 in the example below) represents a basic feature because it is used in all pagination solutions mentioned here. This method accepts search arguments & the pageable instance and provides a result as a SearchHits<T> type. The T defines a type used in our repository (the City class in our case).

The usage is similar to the old searchDeprecated method (see section "Find Cities by Dynamic Query" in my previous article). The main difference is the feature is triggered via an esTemplate instance (line 7) instead of the repository (see line 14). The esTemplate  requires us to specify a document type (City class in our case) as we don't use the repository with such a definition.

@Service
@RequiredArgsConstructor
@Slf4j
public class CityService {

	@NonNull
	final ElasticsearchOperations esTemplate;

	public SearchHits<City> searchHits(String name, String country, String subcountry, 
                                       Pageable pageable) {
		CriteriaQuery query = buildSearchQuery(name, country, subcountry);
		query.setPageable(pageable);

		return esTemplate.search(query, City.class);
	}

	private CriteriaQuery buildSearchQuery(String name, String country, String subcountry) {
		var criteria = new Criteria();
		if (nonNull(name)) {
			criteria.and(new Criteria("name").contains(name));
		}
		if (nonNull(country)) {
			criteria.and(new Criteria("country").expression(country));
		}
		if (nonNull(subcountry)) {
			criteria.and(new Criteria("subcountry").is(subcountry));
		}
		return new CriteriaQuery(criteria);
	}

}

Next, we need to expose this feature on the /search_hits path (line 10)  by the searchHits method in our CityController (lines 11-14) as:

@RequestMapping(value = CityController.ROOT_PATH, produces = APPLICATION_JSON_VALUE)
@RequiredArgsConstructor
public class CityController {

	static final String ROOT_PATH = "/api/cities";

	@NonNull
	final CityService service;

	@GetMapping("/search_hits")
	public SearchHits<City> searchHits(@PathParam("name") String name, @PathParam("country") String country,
			@PathParam("subcountry") String subcountry, Pageable pageable) {
		return service.searchHits(name, country, subcountry, pageable);
	}

}

The endpoint can be verified here. The output should look like this:

{
  "totalHits": 3,
  "totalHitsRelation": "EQUAL_TO",
  "maxScore": "NaN",
  "scrollId": null,
  "searchHits": [
    {
      "index": "city",
      "id": "yqoYEIIB55LQo2aMkOKS",
      "score": "NaN",
      "sortValues": [
        "benešov"
      ],
      "content": {
        "id": "yqoYEIIB55LQo2aMkOKS",
        "name": "Benešov",
        "country": "Czech Republic",
        "subcountry": "Central Bohemia",
        "geonameid": 3079508
      },
      "highlightFields": {},
      "innerHits": {},
      "nestedMetaData": null,
      "routing": null,
      "explanation": null,
      "matchedQueries": []
    },
    ...
  ],
  "aggregations": null,
  "suggest": null,
  "empty": false
}

This approach is fairly easy to implement, but the result doesn't provide enough pagination information (e.g. page number & size, sorting information, etc.). It's impossible to implement the pagination feature properly without this information.

Let's check two other options to retrieve an output with the correct pagination information. 

SearchPage Response

Let's extend our CityService with a searchPage method. Here, we just call the searchHits method described above, but we wrap it with a searchPageFor method from the SearchHitSupport class.

public SearchPage<City> searchPage(String name, String country, String subcountry, Pageable pageable) {
	return SearchHitSupport.searchPageFor(searchHits(name, country, subcountry, pageable), pageable);
}

The searchPageFor method is quite simple. It just re-map our search result defined as SearchHits to SearchPageImpl.

public static <T> SearchPage<T> searchPageFor(SearchHits<T> searchHits, @Nullable Pageable pageable) {
  return new SearchPageImpl<>(searchHits, (pageable != null) ? pageable : Pageable.unpaged());
}

This search feature can be exposed in our controller like this:

@GetMapping("/search_page")
public SearchPage<City> searchPage(@PathParam("name") String name, @PathParam("country") String country,
		@PathParam("subcountry") String subcountry, Pageable pageable) {
	return service.searchPage(name, country, subcountry, pageable);
}

The endpoint can be verified here. The output should look like this:

{
  "content": [
    {
      "index": "city",
      "id": "yqoYEIIB55LQo2aMkOKS",
      "score": "NaN",
      "sortValues": [
        "benešov"
      ],
      "content": {
        "id": "yqoYEIIB55LQo2aMkOKS",
        "name": "Benešov",
        "country": "Czech Republic",
        "subcountry": "Central Bohemia",
        "geonameid": 3079508
      },
      "highlightFields": {},
      "innerHits": {},
      "nestedMetaData": null,
      "routing": null,
      "explanation": null,
      "matchedQueries": []
    },
    ...
  ],
  "pageable": {
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "pageNumber": 0,
    "pageSize": 5,
    "unpaged": false,
    "paged": true
  },
  "searchHits": {
    "totalHits": 3,
    "totalHitsRelation": "EQUAL_TO",
    "maxScore": "NaN",
    "scrollId": null,
    "searchHits": [
      {
        "index": "city",
        "id": "yqoYEIIB55LQo2aMkOKS",
        "score": "NaN",
        "sortValues": [
          "benešov"
        ],
        "content": {
          "id": "yqoYEIIB55LQo2aMkOKS",
          "name": "Benešov",
          "country": "Czech Republic",
          "subcountry": "Central Bohemia",
          "geonameid": 3079508
        },
        "highlightFields": {},
        "innerHits": {},
        "nestedMetaData": null,
        "routing": null,
        "explanation": null,
        "matchedQueries": []
      },
      ...
    ],
    "aggregations": null,
    "suggest": null,
    "empty": false
  },
  "totalPages": 1,
  "totalElements": 3,
  "size": 5,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "first": true,
  "last": true,
  "numberOfElements": 3,
  "empty": false
}

Note: you can see the real content is mentioned twice (under content and searchHits elements).

Page Response

The last option is demonstrated by a search method in our CityService. In this method, we just call the previous searchPage method described above, but we wrap it again with a unwrapSearchHits method from the SearchHitSupport class.

@SuppressWarnings("unchecked")
public Page<City> search(String name, String country, String subcountry, Pageable pageable) {
	return (Page<City>) SearchHitSupport.unwrapSearchHits(searchPage(name, country, subcountry, pageable));
}

The searchPage method is quite complex as it accepts different arguments. We cannot call unwrapSearchHits(searchHits(...)) directly, because it returns the direct content as a List.

Note: you can also construct PageImpl manually and skip all the mentioned helper methods above. It's just a matter of the developer's preference.

This search feature can be exposed in our controller like this:

@GetMapping
public Page<City> search(@PathParam("name") String name, @PathParam("country") String country,
		@PathParam("subcountry") String subcountry, Pageable pageable) {
	return service.search(name, country, subcountry, pageable);
}

The endpoint can be verified here. The output should look like this:

{
  "content": [
    {
      "id": "yqoYEIIB55LQo2aMkOKS",
      "name": "Benešov",
      "country": "Czech Republic",
      "subcountry": "Central Bohemia",
      "geonameid": 3079508
    },
    ...
  ],
  "pageable": {
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "pageNumber": 0,
    "pageSize": 5,
    "unpaged": false,
    "paged": true
  },
  "last": true,
  "totalPages": 1,
  "totalElements": 3,
  "size": 5,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "first": true,
  "numberOfElements": 3,
  "empty": false
}

Conclusion

This article has covered the upgrade to the latest Spring Data Elasticsearch 4.4 with Elasticsearch 7.17 (at the time of the article). Next, we demonstrated three different solutions to retrieve a paginated response from Elasticsearch. 

Personally, I prefer the last option even though it's a little bit complicated. The output is simplest and it contains all expected/needed attributes. The complete source code demonstrated above is available in my GitHub repository.

Please, let me know in the comments if you know a simpler or better solution.