Practical Generators in Go 1.23 for Database Pagination

The recent introduction of range functions in Go 1.23 marks a significant advancement in the language’s capabilities. This new feature brings native generator-like functionality to Go, opening up new possibilities for writing efficient and elegant code. In this article, we will explore range functions and demonstrate their practical application through a real-world example: paginated database queries.

As a software engineer, I've experienced the critical importance of efficient data handling, especially when working with large datasets and performance-intensive applications. The techniques discussed here have broad applications across various domains, helping to optimize system resource usage and improve overall application responsiveness.

Understanding Database Pagination

Database pagination is an essential technique for efficiently managing large datasets by retrieving data in smaller, manageable chunks. This approach helps reduce memory consumption and optimizes resource utilization, making it particularly useful in applications where handling large amounts of data without overloading system resources is crucial.

In this article, we’ll use Postgres as our database, leveraging its Cursors feature for efficient query result pagination. Here’s a brief overview of the syntax:

BEGIN;

DECLARE my_unique_cursor CURSOR 
FOR 
    SELECT ... FROM ... WHERE ... ORDER BY ...
    
FETCH 42 FROM my_unique_cursor; // Return the first 42 rows
FETCH 24 FROM my_unique_cursor; // Return the second 24 rows

ROLLBACK;

This approach allows us to declare a unique cursor for our query and then fetch rows in batches, providing a foundation for efficient data retrieval.

Go 1.23 Generators: A New Paradigm

Go 1.23 introduces range functions, a feature that simplifies iteration over custom data types and sets. To implement this functionality, we need to create a custom function that returns one of the following signatures:

func(yield func() bool)
func(yield func(V) bool)
func(yield func(K, V) bool)

For a comprehensive understanding of range functions, refer to the official Go documentation.

Implementing Pagination With Go 1.23

Let’s examine a practical implementation of pagination using Go 1.23’s new features. We’ll start with a basic database schema:

CREATE TABLE test (
    id SERIAL PRIMARY KEY,
    text VARCHAR(255) NOT NULL
);

INSERT INTO test (text)
VALUES
    ('row 0'),
    ('row 1'),
    -- ... more rows ...
    ('row 10');

Now, let’s define our Paginate function:

func Paginate[T any](
    ctx context.Context,
    db *sql.DB,
    query string,
    batchSize int,
    decoder Decoder[T],
) (func(func(T, error) bool), error)

Here, Decoder[T] is defined as type Decoder[T any] func(rows *sql.Rows) (T, error), allowing for type-safe decoding of database rows.    

The implementation of Paginate demonstrates the power of Go 1.23’s range functions:

func Paginate[T any](
    ctx context.Context,
    db *sql.DB,
    query string,
    batchSize int,
    decoder Decoder[T],
) (func(func(T, error) bool), error) {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return nil, fmt.Errorf("error starting transaction: %w", err)
    }

    cursor, err := NewRandomCursor()
    if err != nil {
        return nil, fmt.Errorf("error generating cursor: %w", err)
    }

    return func(yield func(T, error) bool) {
        defer func() {
            _ = tx.Rollback()
        }()

        _, err = tx.ExecContext(ctx, fmt.Sprintf("DECLARE %s CURSOR FOR %s", cursor, query))
        if err != nil {
            log.Printf("Error declaring cursor: %v", err)
            return
        }

        for {
            page, ok, err := ReadPage[T](
                func() (*sql.Rows, error) {
                    return tx.QueryContext(ctx, fmt.Sprintf("FETCH %d FROM %s", batchSize, cursor))
                },
                decoder,
            )
            if err != nil {
                var unit T
                yield(unit, err)
            }

            for _, row := range page {
                if !yield(row, nil) {
                    return
                }
            }

            if !ok {
                break
            }
        }
    }, nil
}

This implementation efficiently manages database transactions, cursor handling, and error management while utilizing Go 1.23’s new range functions.

Practical Application

To demonstrate the practical use of our pagination function, consider the following main function:

func main() {
    // Assume proper context and database setup
    
    query := "SELECT id, text FROM test ORDER BY id"
    batchSize := 2

    pagination, err := Paginate[Entry](ctx, db, query, batchSize, DecodeEntry)
    if err != nil {
        log.Fatal(err)
    }

    for row, err := range pagination {
        if err != nil {
            log.Printf("Error fetching rows: %v", err)
            return
        }

        log.Printf("Row: %v", row)
    }
}

Executing this code yields results that clearly illustrate the pagination process:

Query 1
Row: {1 row 0}
Row: {2 row 1}
Query 2
Row: {3 row 2}
Row: {4 row 3}
// ... subsequent queries and rows ...

Conclusion

The introduction of range functions in Go 1.23 represents a significant step forward in the language’s evolution. This feature enables developers to create more efficient and readable code, particularly when dealing with large datasets and complex data structures.

In the context of database pagination, as demonstrated in this article, range functions allow for the creation of flexible and reusable solutions that can significantly enhance performance and resource management. This is particularly crucial in fields like fintech, where handling large volumes of data efficiently is often a key requirement.

The approach outlined here not only improves performance but also contributes to better code maintainability and readability. As Go continues to evolve, embracing features like range functions will be crucial for developers aiming to write more expressive and performant code.

I encourage fellow developers to explore the full potential of Go 1.23’s range functions in their projects. The possibilities for optimization and improved code structure are substantial and could lead to significant advancements in how we handle data-intensive operations.