Building a Reddit Scraper in Go: JSON Listings, Pagination, Rate Limits, and Where It Breaks

TL;DR

Reddit's public .json listings let you scrape a subreddit in about 100 lines of Go. You decode the listing envelope, follow the after cursor, send an honest User-Agent, and back off on HTTP 429. There are two hard limits: a listing stops at roughly 1,000 items, and requests from cloud or datacenter IPs often get a flat 403. For anything in production, use the official OAuth API or a managed endpoint.

Reddit is one of the richest public sources of opinion data on the web. Product teams mine it for feature requests, marketers track brand mentions, and ML engineers build datasets from it. Since the 2023 API pricing changes, "just use the API" has become a harder sell for small projects, so many developers go back to scraping.

This article walks through a small, dependency-free Go scraper that exports a subreddit's newest posts to CSV. It also covers the failure modes you'll actually hit.

Compliance first. Read Reddit's User Agreement and Data API Terms before collecting anything. Collect only public data, keep request rates low, don't re-identify users, and don't use Reddit content to train models without the appropriate licence. Commercial use generally requires an agreement with Reddit.

How Reddit Exposes Data Without an API Key

Add .json to almost any Reddit listing URL and you get the same data the old web UI used:

https://www.reddit.com/r/golang/new.json?limit=100&raw_json=1

• limit goes up to 100 items per page.
• raw_json=1 stops Reddit from HTML-escaping &, < and > in titles.
• after is the pagination cursor (a "fullname" like t3_1abc23).

The response is a Listing envelope:

{
  "kind": "Listing",
  "data": {
    "after": "t3_1abc23",
    "children": [
      { "kind": "t3", "data": { "id": "1abc23", "title": "...", "score": 42, ... } }
    ]
  }
}

You only need to model the fields you use. Go's encoding/json ignores the rest, which makes the scraper resilient when Reddit adds fields.

The Scraper

package main

import (
    "encoding/csv"
    "encoding/json"
    "fmt"
    "net/http"
    "os"
    "strconv"
    "time"
)

type listing struct {
    Data struct {
        After    *string `json:"after"`
        Children []struct {
            Data post `json:"data"`
        } `json:"children"`
    } `json:"data"`
}

type post struct {
    ID          string  `json:"id"`
    Title       string  `json:"title"`
    Author      string  `json:"author"`
    Score       int     `json:"score"`
    NumComments int     `json:"num_comments"`
    CreatedUTC  float64 `json:"created_utc"`
    Permalink   string  `json:"permalink"`
}

var client = &http.Client{Timeout: 15 * time.Second}

func fetchPage(sub, sort, after string) (*listing, error) {
    url := fmt.Sprintf("https://www.reddit.com/r/%s/%s.json?limit=100&raw_json=1", sub, sort)
    if after != "" {
        url += "&after=" + after
    }
    for attempt := 0; attempt < 4; attempt++ {
        req, _ := http.NewRequest("GET", url, nil)
        // Reddit blocks generic user agents. Identify your script honestly.
        req.Header.Set("User-Agent", "go:subreddit-archiver:v0.1 (by /u/your_username)")
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= 500 {
            resp.Body.Close()
            wait := time.Duration(1<<attempt) * 5 * time.Second
            fmt.Fprintf(os.Stderr, "status %d, backing off %s\n", resp.StatusCode, wait)
            time.Sleep(wait)
            continue
        }
        if resp.StatusCode != http.StatusOK {
            resp.Body.Close()
            return nil, fmt.Errorf("unexpected status %d", resp.StatusCode)
        }
        var l listing
        err = json.NewDecoder(resp.Body).Decode(&l)
        resp.Body.Close()
        return &l, err
    }
    return nil, fmt.Errorf("gave up after retries")
}

func main() {
    sub := "golang"
    if len(os.Args) > 1 {
        sub = os.Args[1]
    }
    w := csv.NewWriter(os.Stdout)
    defer w.Flush()
    w.Write([]string{"id", "created", "score", "comments", "author", "title", "url"})

    seen := map[string]bool{}
    after := ""
    for page := 0; page < 10; page++ { // listings stop around 1,000 items
        l, err := fetchPage(sub, "new", after)
        if err != nil {
            fmt.Fprintln(os.Stderr, "error:", err)
            return
        }
        for _, c := range l.Data.Children {
            p := c.Data
            if seen[p.ID] {
                continue
            }
            seen[p.ID] = true
            w.Write([]string{
                p.ID,
                time.Unix(int64(p.CreatedUTC), 0).UTC().Format(time.RFC3339),
                strconv.Itoa(p.Score),
                strconv.Itoa(p.NumComments),
                p.Author,
                p.Title,
                "https://www.reddit.com" + p.Permalink,
            })
        }
        if l.Data.After == nil {
            break
        }
        after = *l.Data.After
        time.Sleep(2 * time.Second) // stay well under the unauthenticated limit
    }
}

Run it:

go mod init redditscraper
go run . golang > golang.csv

Design Notes

• After *string: after is null on the last page. A pointer tells "no more pages" apart from an empty string.
• Deduplication: on /new, posts can shift between pages while you paginate, so the same ID can show up twice. A map[string]bool is enough for one run. For recurring jobs, persist IDs (SQLite with id as the primary key works well).
• created_utc is a float: decode it as float64. Decoding into int64 fails on some responses.
• Streaming CSV: rows are written as they arrive, so memory use stays flat no matter how big the run is.

Where It Breaks

This is the part most tutorials skip.

1. The 1,000-item ceiling. Every Reddit listing (new, top, hot, search) stops paginating at about 1,000 items. You cannot backfill a busy subreddit's full history from listings. To keep a complete archive, run the scraper on a schedule (for example, every 15 minutes against /new) and accumulate results yourself.

2. 403s from cloud IPs. When I ran the code above from a cloud VM, Reddit returned 403 Forbidden on the first request, even with a descriptive User-Agent. The same code typically works from a residential connection. Reddit filters unauthenticated traffic from datacenter ranges aggressively, so a scraper that works on your laptop can fail as soon as you deploy it. Retrying a 403 doesn't help, which is why the code only retries 429 and 5xx.

3. Rate limits. Unauthenticated access is throttled hard, and the limits aren't published. Watch for 429, and when they're present, respect the x-ratelimit-remaining and x-ratelimit-reset headers. A 2-second delay between pages is a conservative starting point.

4. Comments are a tree, not a list. /comments/<id>.json returns a two-element array (the post, then the comment tree). Deep threads are truncated with "kind": "more" stubs that need extra requests to expand. Budget for this: it's usually where most of your request volume goes.

5. Deleted content. Authors show up as [deleted] and bodies as [removed]. Decide up front whether to keep those rows.

The Official Route: OAuth

For a sustained, deployable workload, register a "script" app at reddit.com/prefs/apps, use the client-credentials flow to get a bearer token, and send requests to https://oauth.reddit.com instead of www.reddit.com. The listing JSON is the same, so the structs above can be reused unchanged. You get documented rate limits and a far lower chance of 403s. The trade-off is Reddit's API terms and pricing for commercial or high-volume use.

The managed route

If you'd rather not run proxies or babysit tokens, a hosted endpoint can return the same data as normalized JSON. With ScrapeAtlas*, for example, the subreddit listing above becomes a single authenticated GET:

req, _ := http.NewRequest("GET",
    "https://api.scrapeatlas.com/v1/reddit/subreddit?subreddit=golang&sort=new", nil)
req.Header.Set("x-api-key", os.Getenv("SCRAPEATLAS_API_KEY"))

The response carries a posts array (id, title, author, score, num_comments, created_utc, url, ...), an after cursor for the next page, and a coverage object that reports whether the result is complete. Optional since/until parameters bound the time window. The same API has endpoints for post comments (/v1/reddit/post/comments) and global search (/v1/reddit/search). The post struct above maps onto that response almost one-to-one, so switching backends is mostly a URL change.

Choosing an approach

Conclusion

Go is a great fit for this job: static binaries, a solid standard-library HTTP client, and encoding/json that tolerates schema drift. The scraping itself is easy. The real engineering work is in handling limits: the 1,000-item ceiling, IP reputation, and rate limits. Design for those from day one and your scraper will keep running after you deploy it.

*I build it.