From Big Data to Agents: My Decade Building Systems

When a Simple Scraper Changed Everything

My first “real” task wasn’t glamorous: scrape some data. We all start at the bottom, and scraping is a rite of passage for data engineers — turning unstructured sources into structured signals. That first job taught me a lesson I’ve carried since 2014/15: access to the right data beats cleverness — and the right pattern beats the shiniest tool.

Looking back at that moment — and the path since — this article traces how I entered the data world and what changed up to today’s AI revolution.

I’ll frame the journey in three waves:

• Big-Data Discovery
• Make the Data Speak
• From Answers to Actions

How did each wave change my view of turning ideas into working systems?

Wave #1: Big-Data Discovery

If I had to pick a starting point, it would be 2014/2015: a younger me chasing a scholarship to get a first foothold in working life. By luck — or misfortune, depending on the day — I landed in a consultancy working on big data projects. I had no idea what any of it meant or how to start, but I’ve always liked a challenge; learning is in my DNA.

That was my first contact with Hadoop jobs, clusters, MapReduce, and writing SQL to make data talk. I didn’t know it then, but this would become the seed of my entire journey into the data world.

In parallel, I explored the data-science path. My final project and master’s thesis focused on sentiment and emotion analysis. That experience helped me understand other needs and “wear other shoes.” Still, my center of gravity was data engineering: how to extract, store, process, and — on the last mile — turn data into actionable insights.

We also have to remember the context. At that time, everything was labeled big data — even when it wasn’t. “Big data” became the magic word to sell innovation. Every company wanted it; very few were ready. Two factors made adoption hard (as we’re seeing now with AI): (1) they didn’t understand the why or the how, and (2) they didn’t make the right culture shift.

The second point matters most. A manager once told me, “You can build a Ferrari, but if people don’t even know what the wheels are, it won’t be used.” The nightmare for any engineer is to pour their soul into something that no one uses. That thought still keeps me up at night. And beneath it sat the real question: who are “they”? Who are the users, what do they need, and how do they actually work?

I remember building an end-to-end system — scrape → transform → visualize in a very fancy way. I was proud of the learning and the build. But there was one painful truth: they weren’t using it. “They” were the end users — business stakeholders and external users. In the excitement of shipping the full system, I hadn’t included them early, anchored the work in their pain points, or validated how they would adopt it.

That hard lesson propelled me to the next wave: Make the Data Speak. How do I involve users from day one and craft a narrative that makes the data impossible to ignore?

Wave #2: Make the Data Speak

Who doesn’t love a good story? I’ve always been a book devourer — chasing narratives I can sink my teeth into. I first heard the term data storytelling when I joined a data-focused company. Before that, I’d worked in sectors where data and technology weren’t the core of the business.

That’s when it clicked: data carries a story. It’s not about decorating dashboards with charts and shipping them. It’s about understanding why we see these values, how they were extracted, and what they imply for a decision. Good storytelling turns raw numbers into a sequence of questions and answers: What changed? Compared to what? Why now? What should we do next?

I also learned that storytelling starts before plotting any graph:

• Audience: who will use this and how will they act on it?
• Context: definitions, lineage, sampling, and caveats—plainly stated.
• Narrative arc: setup (baseline), conflict (the deviation), resolution (recommendation).
• Decision hook: the one action this view is meant to unlock.

The result was a shift in how I built: scrape → transform → explain. I involved users earlier, validated their questions, and designed visuals that made variance, uncertainty, and deltas obvious. The win wasn’t a prettier chart; it was adoption—people trusted the pipeline and used it to decide.

Reframing data as a story changed my work: once you can state the question, show provenance, and drive a decision, the next step is obvious—let the system do part of the work. That curiosity pulled me into AI: first to compress the storytelling loop (LLMs), and then to close it with actions (agents).

Wave #3 (2024–Now): From Answers to Actions

Over dinner with a good friend, we talked about how AI was suddenly able to handle real tasks — simply and reliably. His comment stuck with me: the barrier to entry was gone. Previously, entering data science required wrestling with heavy frameworks and infrastructure. Today, AI lowers that bar: for many small, practical applications, you can go from idea to working prototype with only a few lines.

I had already used AI for small things — throwaway scripts, summarizing endless PDFs — but on his recommendation, I decided to go deeper. That decision changed how I saw AI.

My first bet was to automate a monotonous task: scraping data. In essence, turn unstructured pages into structured records. Within hours—and with relatively simple code — I could scrape straightforward sites and dump clean outputs. It was a game changer: not because scraping was new, but because the path from idea to working system had collapsed.

Once I started digging deeper, there was no going back. I began seeing practical applications everywhere — and ways to fix old pain points I’d run into.

Later, I changed jobs and moved to a very different stack. I asked myself whether I could take on the challenge without slowing the team and still contribute quickly. With a solid foundation, AI removed most of that barrier: what I didn’t know, I could learn in context; what I couldn’t automate, I could scaffold. Within weeks, I was shipping useful, well-structured code in a new language — AI drafted scaffolds, and I iterated: reviewing, asking targeted questions, and validating behavior until the design felt native. In practice, AI turned learning and building into one loop.

Final Thoughts

There are three things I want to underline:

1. Data is still the central pillar of every “revolution.” We’ve generated a massive amount of data that no team can fully read or understand end-to-end. What we can do is detect patterns and make searching that ocean efficient, without building bespoke solutions for every single use case. And yes — the more relevant data we feed an assistant or an agent (what everyone now calls context), the easier it becomes for it to make the right call.
2. AI doesn’t solve every problem — and not every problem should be solved with AI. This has two parts. First, AI won’t magically fix the complexity of a broken solution. We often imagine AI as Mr. Wolf from Pulp Fiction — “I solve problems.” — showing up to handle everything, but it’s not like that. Many times, you have to look at the root cause and set solid foundations so AI can actually add useful value. Second, there’s a trend I keep seeing in conversations with friends: if a project doesn’t include AI (just like the big data days), it doesn’t feel “innovative.” That may be the perception, but it doesn’t mean AI is the right tool. Sometimes the simplest non-AI solution is the one that truly works.
3. Culture sets the path. Tools don’t adopt themselves. Companies need to lead a cultural shift: make problems explicit, create safe sandboxes, reward small wins, and train people to think in data and systems. When leadership, incentives, and learning align, the organization can keep up with how fast the world is changing — and actually benefit from it.

In closing: Keep data at the center, be honest about the problem you’re solving, and use AI where it clearly adds leverage — no more, no less.