The Bottleneck of Scaling

Any input/output operation, be it accessing a file, handling an HTTP request, or a database connection, is based on 3 fundamental system concepts — file descriptors, kernel memory, and heap size.

This article discusses how modern languages help developers handle behind-the-scenes file descriptor, kernel memory, and heap management. These three concepts are major bottlenecks for scaling.

1. File Descriptors

A file descriptor is just a positive number that is used by the kernel to identify any open input/output stream or connection. It is defined by the kernel for a process. The following file descriptors are defined by default for a process:

• 0 – Standard Input (stdin)
• 1 – Standard Output (stdout)
• 2 – Standard Error (stderr)

Any subsequent I/O operation gets the next available integer as file-descriptor. The file descriptor value can be adjusted by using the ulimit -n command in Linux. Each application, whether it is a web server written in Java Spring Boot, an API server written in Go using net/http and gorilla-mux, or a Python Flask app, is a single process. Each process has only 1024 file descriptors defined by default. That means each application can perform only 1024 I/O operations simultaneously. This seems like an amazing concept when we talk about scaling our application or API server. As many times as we come across this question — how can we scale our API server or web application to handle 100k or 1 million requests per second?

This is where our modern languages play their role very beautifully behind the scenes to enable developers to develop the application to handle such scale.

2. Kernel Memory

At a lower layer than file descriptors, when an incoming TCP connection hits the network card, the Linux kernel performs a 3 Way TCP handshake for that connection. The handshake lifecycle includes the states: SYN -> SYN-ACK -> ACK. The number of requests equal to the defined file descriptor value are processed immediately, assigned a file descriptor, and forwarded to the application for further processing. When FDs are exhausted, the Kernel maintains a queue for requests waiting for FDs to become available so your application can process them. 

The same thing happens when a request is processed, and the response is ready to be sent back to the client. This queue is maintained within RAM by read buffers(rmem) and write buffers(wmem). The size of buffers is defined in memory by the kernel and is dynamic, depending on network throughput, round-trip time, and memory pressure.

The kernel network memory is non-paged, i.e cannot be swapped to disk. It’s a big bottleneck as it directly depends on physical memory. For example, if there are 100,000 open connections and each connection holds an average of 128KB of kernel memory, it comes to 12.8GB of physical RAM. This is clearly a kernel overhead, and it doesn’t show up in JVM heap metrics or Go runtime statistics. rmem and wmem buffers are governed by kernel parameters defined in /proc/sys/net/ipv4/

3. Heap Size

When TCP connections are assigned file descriptors and kernel memory is reserved, they enter user space, which is the memory managed by the application runtime — Java JVM, Node.js V8 Engine, Python interpreter, Go runtime, etc.

Each connection stores objects in the heap within three categories:

1. Connection metadata – Keep-alive timers, IP State, Socket Wrappers, etc.
2. Cryptographic session context – handshake caches, cipher states, TLS/SSL keys, etc.
3. Serialized payload buffers – response queues, JSON strings, ORM entity maps, etc.

A connection that is encrypted via TLS takes a lot more space in the heap compared to a regular connection. For an encrypted connection, the application has to save symmetric keys, cipher contexts, session tickets, etc. onto the heap.

A regular TCP socket object in the heap consumes 2KB to 5KB of space, whereas a TLS 1.3 socket object consumes 20KB to 100KB of heap space. If an API maintains 10,000 idle TLS connections, it will consume 200MB to 1GB of heap space.

When an application runs, the runtime asks the kernel for memory space as the application creates objects. The application keeps creating objects, and the kernel keeps reserving memory for those objects; this is called the heap. The maximum heap size can be defined by different programming languages at runtime; for example, in Java, -Xmx4g reserves 4GB for the heap. The operating system promises to provide that much memory as heap space for the application, but it doesn’t reserve it all at once. As the application creates objects, the kernel continues to reserve memory. When objects are marked as done, the garbage collector removes them from the heap. 

When an incoming request hits our API server, the application uses heap space to convert raw bytes to the application-specific data structure. Once the application finishes processing the request and returns the response, those objects in the heap become unreachable or dead. When the garbage collector sweeps those objects to reclaim that memory, it doesn’t return the memory immediately; instead, the JVM or Go runtime keeps that freed memory in its internal pool. If a new HTTP request arrives within 1 millisecond, the runtime assigns the required memory from the free memory in the pool.

Now imagine 10,000 new requests arriving at the same time, each with 2MB of raw bytes, and the runtime trying to allocate heap for the objects; the app instantaneously uses 20GB of memory. This is called GC thrashing, as the runtime rapidly creates required objects in the heap faster than the GC can clean them. The garbage collector is an application thread itself; when the heap gets 80%-90% full, the garbage collector panics and consumes 100% of CPU cores to scan millions of memory pointers to find dead objects. The runtime, like the JVM or Node.js garbage collector, may stop other code execution while it reorganizes the memory. 

So, how do runtimes like Go and the JVM handle GC thrashing? Go follows a simple strategy – avoid creating objects on the heap. The fastest GC collector is the one that has nothing to collect. The Go compiler compiles the application to see if variables outlive their functions. If a struct is used only inside a function, Go pushes the struct to the stack instead of the heap, and the stack pointer just drops when the function returns. The memory is reclaimed in 1 CPU cycle without even involving the garbage collector.

If Go does have to clean the heap, its GC runs concurrently along with other goroutines and is broken into several micro pauses. Go provides sync.Pool to help developers to reuse heap memory while creating objects. For example to instead of creating millions of []bytes for JSON parsing for every new request, developers can use sync.Pool as follows:

// Instead of creating a new buffer for every HTTP request:
var bufferPool = sync.Pool{
    New: func() any { return new(bytes.Buffer) },
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
    buf := bufferPool.Get().(*bytes.Buffer) // 1. Grab an existing buffer from pool
    buf.Reset()
    defer bufferPool.Put(buf)               // 2. Put it back when done!

    // Parse JSON into 'buf' without allocating new heap memory
}

By recycling buffers via sync.Pool, high-concurrency APIs can handle 100,000 requests/sec with near-zero new heap allocations.

Java takes a different approach. Because Java applications historically create millions of short-lived objects on the heap, the JVM relies on Generational Hypotheses and Generational Collectors (like G1GC, ZGC, and Shenandoah). G1GC can be used like java -XX:+UseG1GC while running Java applications. G1GC divides the Heap memory into physical regions: Young Generation (Eden & Survivor spaces) and Old Generation. It kind of sorts objects into different regions so that it doesn't have to scan the complete heap and can clean where most of the marked objects live. We can also mention -XX:MaxGCPauseMillis=200 to tell G1 to pause the application for no more than 200ms, but this is not guaranteed.  

Older JVM collectors like Parallel GC used to freeze the entire application to clear the heap when full, leading to multi-second latency spikes. Modern JVMs introduce ZGC (Z Garbage Collector) and Shenandoah. ZGC uses specialized CPU pointer references to track moved objects in real time. ZGC can clean, move, and compact terabytes of heap memory concurrently while your API requests are actively running. ZGC guarantees GC pause times under 1 millisecond, regardless of whether your heap is 500 MB or multi-terabytes.

Conclusion

Keep track of these three core concepts — file descriptors, kernel memory, and heap size to know when to scale.

1. File Descriptor Saturation Signals

File descriptors represent the system's open handles. When an application hits its FD threshold, the operating system stops accepting connections. The following are example scenarios that indicate when to scale. 

• Check Kernel-wide statistics from /proc/sys/fs/file-nr, per process fds - /proc/<pid>/fd, Prometheus exposes process_open_fds. If it consistently breaches the 80–85% threshold, it's time to scale. 
• You have already tuned ulimit -n and LimitNOFILE up to standard safety thresholds (e.g., 65,536 or 104,857), but process FD counts continue climbing toward the max. 
• Network interfaces show growing SYN-to-LISTEN socket counts and drops in netstat -s under the listen queue overflow metric.

2. Kernel Memory Pressure Signals

Because TCP receive (rmem) and transmit (wmem) buffers are non-paged, they cannot overflow onto disk swap. When kernel network memory fills up, the OS drops packets. Below are the scenarios related to kernel memory breach.

• Check /proc/net/sockstat under TCP: inuse and matching /proc/sys/net/ipv4/tcp_mem thresholds. 
• Netstat counters (netstat -s | grep -i retrans) show a sharp rise in TCP Retransmission rates (>1–2%). 
• Latency spikes occur because the kernel is dynamically shrinking socket buffers down to tcp_rmem minimums (4 KB) to avoid running out of physical RAM, throttling TCP window sizes.

3. Heap Size & Garbage Collection (GC) Thrashing Signals

When user-space heap allocations outpace the garbage collector's ability to sweep dead objects (like parsed JSON payloads or session states), application performance collapses. The runtime (JVM or Go) spends more than 15–20% of its total CPU time running GC sweeps (go_gc_cpu_fraction or JVM GC CPU utilization). 

In Go, metrics show the pacer triggering Mark Assist, stealing CPU time from worker goroutines to help clean up memory. You can check the runtime package /cpu/classes/gc/mark/assist:cpu-seconds metrics to see if GC is asking for more help from CPU. In Spring Boot, you can use Actuator and Micrometer to expose relevant endpoints to monitor the threshold values.