A Day in the Backend

The company itself seemed like your average run-of-the-mill startup: serious-looking people quietly swearing at their screens (which to my horror, displayed Linux prompts), punctuated only by the occasional scream from the direction of the QA group.

I was assigned to the XDB team, which stands for Xeround Data Blade –  our distributed, scalable storage backend. The XDB is responsible for distributing and storing data on multiple backend servers, while maintaining high availability. It also has the ability to adapt its size (internally as well as externally – by throwing additional blades into the mix) to user requirements, so that at the end of the day, we don’t use more system resources than we actually need, and users are only billed for what they use.

How hard can it be? I asked myself. To be fair, they did let me read documents for a week or two, but then I was thrown head-first into an overwhelming mass of “under the hood” low level code and complex distribution algorithms.

Luckily, being an experienced startup veteran, I was equipped with all the necessary tools to cope with the steep learning curve ahead: caffeine, a sleeping bag and a shovel. Don’t ask me about the shovel.

Gradually I was sucked in, discovering never ending layers of depth and challenge in the code, from the very lowest levels of data compression and memory optimizations, to relatively high level concepts (remember, this is the backend; what is high level to us is, to most, an obscure implementation detail) such as data distribution, consistency and transactionality (is that even a real word?).

An even greater challenge for me is to make the connection between my day to day tasks and user level SQL queries. Since the user’s queries are highly compressed and optimized by our front end, what we end up getting in the backend sometimes reminds me of the Matrix movie- you’d have to be Neo to see the SQL through all those bits and bytes.

Keeping the user’s perspective in mind can be incredibly challenging when you’re writing code to reduce memory fragmentation or digging into a core file to find out what went wrong – but precisely here lies the greatest motivation – in the end, all those nights of debugging and days of coding, all that haggling with the product people and stealthy tiptoeing by the QA room, all those things translate into something that someone, somewhere, is using to store data that is actually important to them! And as you know, there is no greater satisfaction for a software engineer than to know that their code is being used in the real world. OK, maybe becoming ludicrously rich, but you know what I mean.. :)