DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Python Async/Sync: Advanced Blocking Detection and Best Practices (Part 2)
  • Python Async/Sync: Understanding and Solving Blocking (Part 1)
  • How To Scale Your Python Services
  • What Does Synchronization With Asyncio Look Like

Trending

  • Beyond @Transactional: Solving the Dual-Write Problem in Distributed Microservices
  • The Five Stages of AI Adoption: A Practical Maturity Model for Software Teams
  • 9 Practical Ways to Fix Laravel Performance Issues in Production
  • Idempotency Under Concurrency: Why Your Application Check Isn't the Last Line of Defense

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Coding
  3. Languages
  4. Python Asyncio: Understanding Concurrency With a Scraper

Python Asyncio: Understanding Concurrency With a Scraper

Use asyncio.create_task() for I/O concurrency, gather or wait to manage task batches, and a Semaphore to prevent resource exhaustion.

By 
Iyanuoluwa Ajao
·
Nov. 18, 26 · Tutorial
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
88 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Introduction 

If you need to scrape a few thousand job postings from a careers page, one way to do it is by using a for loop, one request after another:

Python
 
import requests

def scrape_all(urls):
    pages = []
    for url in urls:
        response = requests.get(url)   # blocks here until the page comes back
        pages.append(response.text)
    return pages

(The real scraper uses a headless browser rather than requests, but the shape is identical.)


This works, but it's slow. For example, if a page takes two seconds to come back, then 1500 pages will take 3000 seconds (50 minutes). Most of that time is spent not computing anything. The program is sitting around waiting for responses to cross the network. While it waits on page 1, it can't even ask for page 2.

Python has more than one answer to "how do I stop wasting that wait?".  The first is concurrency, while the second is parallelism. 

Concurrency vs Parallelism

The analogy that explains the difference is when you have two YouTube videos open in two browser tabs, and both are buffering.

  • Concurrency is watching tab 1, having it stall to buffer, switching to tab 2 while tab 1 catches up, then switching back once tab 1 is ready. You're still only watching one video at any given instant, but nothing sits idle for long, and the moment one thing is blocked, your attention moves to whatever isn't.
  • Parallelism is watching both videos at the same time, side by side, with two separate screens and two separate sets of eyes. No switching and no waiting on one for the other. They're happening simultaneously.

Concurrency is about overlapping waiting, and it works even on a single CPU core. Most of that "watching" time is actually idle time. Parallelism is about overlapping working, and it requires multiple CPU cores doing computation at the same moment. You can't parallelize your way out of a slow network connection, but you should be able to get better at not wasting the wait.

That maps directly onto the two categories of workload:

  • CPU-bound work (image processing, number crunching, data transformation) is limited by how fast the processor can compute. This is where parallelism helps since what you need are more cores, more simultaneous computation. Python's tool for this is the multiprocessing module. It sidesteps the GIL by running separate processes with separate interpreters.
  • I/O-bound work (network requests, waiting on an LLM API,  database queries) is limited by how long you sit around waiting for something external to respond. This is where concurrency helps because while one request is "buffering," you can be making progress on another. There are two different tools for working with concurrency. The first is called threading, while the second is called asyncio.

Why Async Fits I/O

This article is about asyncio. It uses cooperative, single-threaded concurrency. In this case, your code explicitly says "I'm waiting, someone else can go now" via await, instead of the OS pre-emptively switching between threads for you. For a scraper, asyncio is a better fit because you are short on patience while 1,500 HTTP requests are all "buffering" at once.

asyncio lets a single thread juggle thousands of these waiting operations concurrently. While one task waits for a response from OpenAI, another can wait for a Playwright page load. Neither is burning CPU time while it waits, since it's just parked, and the event loop wakes up the thread the moment its data arrives. Nothing is happening in parallel because there's still only one thread, but most of the waiting is not wasted.

This article explains asyncio's concurrency primitives by building a real scraper. Given a company's careers page, it:

  1. Loads the page and collects the URL of every job posting. For each of those URLs, loads the posting and asks OpenAI to pull out structured fields.

Step 2 is where all the waiting happens. The reason is that it's the same two slow operations repeated for every URL. These two are the only pieces of the scraper you need to know about:

Python
 
async def scrape_webpage(url: str) -> list[str]:
    """Loads a page in a headless browser (Playwright) and returns its rendered HTML."""

async def extract_job_information(html_document: str, apply_url: str) -> dict:
    """Sends the page text to OpenAI and returns structured fields."""


A call to the second one returns a dictshaped like this: 

Python
 
{
    "job_title": "Senior Data Engineer",
    "company_name": "Example Corp",
    "location": "Remote - US",
    "commitment": "full-time",
    "seniority_level": "Senior Level",
    "job_description": "We're looking for ...",
    "apply_url": "https://example.wd5.myworkdayjobs.com/...",
}


This article explains how to run 1,500 job URLs effectively. 

Python
 
async def process(urls: list[str]) -> list[dict]:
    ...  # scrape and extract every URL, in whatever way this strategy chooses


The Basics

There are three concepts to understand. There are coroutines (async def and await), the event loop, and the difference between await and asyncio.create_task(). The examples below use asyncio.sleep() to stand in for a network request. They run as-is. You can paste them into a file and try them. After each one, we'll connect it to a real scraper.

Coroutines: async def and await

Python
 
import asyncio

async def fetch(name, seconds):
    print(f"{name}: start")
    await asyncio.sleep(seconds)   # stands in for a network request
    print(f"{name}: done")
    return name


Three rules cover almost everything you'll run into:

1. async def defines a coroutine function. It looks like a normal function, but it's allowed to pause partway through.

2. Calling it does not run it. You get back a coroutine object.

Python
 
print(fetch("a", 1))
# <coroutine object fetch at 0x7f8c358a39f0>
# RuntimeWarning: coroutine 'fetch' was never awaited


You can see that a: start never printed. The warning "was never awaited" means you called the function and forgot to await it, so nothing ran.

3. await runs a coroutine and gives you its result. It can only be used inside another async def. asyncio.run()starts the whole chain from regular code. 

Python
 
asyncio.run(fetch("a", 1))
# a: start
# a: done


await does two jobs at once. First, it runs the coroutine and hands back the return value, like a normal function call. Second, it marks a point where this function is willing to pause: "I'm waiting on something slow, so someone else can run while I wait". That second job is the entire reason asyncio exists.

The scraper is built from exactly this. The two awaits are the pause points. The first awaitloads the page while the second await calls the API. 

Python
 
async def _scrape_and_extract(url: str) -> dict:
    documents = await scrape_webpage(url)
    html = remove_html_tags(documents[0])
    return await extract_job_information(html_document=html, apply_url=url)

The scrape_webpage returns a list because it can also follow pagination, and a single job posting is one page; hence documents[0]. And remove_html_tags is an ordinary, non-async helper that strips the markup so the model sees plain text. It has nothing to do with concurrency, so you can ignore it from here on.

The Event Loop: Who Decides What Runs

When a function pauses at an await, "someone else can run" needs an event loop. From the browser tabs analogy earlier, the event loop is you. Each coroutine is a tab. An await is the moment a tab starts buffering, and the event loop is what notices and flips to a tab that's ready.

The event loop keeps track of which tasks are ready to run and which are waiting on something (a response, a timer). It runs one ready task until that task hits an await on something that isn't finished yet, sets it aside, and picks the next ready one. When whatever it was waiting for completes, the task goes back on the ready list. All of this happens on a single thread. Only one piece of your code is running at any instant. Concurrency comes from the waiting overlapping, not from the code running simultaneously.

There are two things to consider: 

  • Switching only happens at an await. Nothing interrupts a task midway. So if a coroutine calls something slow without awaiting it (e.g., time.sleep(2) or requests.get(url)), the loop can't switch away, and every other task pauses until that call returns. Inside async code you use await asyncio.sleep(2) and async APIs (Playwright's async API, LangChain's ainvoke) instead of their blocking equivalents. This is why the scraper uses async_playwright rather than a plain HTTP library.
  • You rarely create the loop yourself.asyncio.run() creates one for a script. In a FastAPI app like this one, uvicorn creates it and runs every request handler as a task on it.

await vs. asyncio.create_task(): Sequential vs Concurrent

It is important to understand that writing await does not make anything concurrent by itself.

Python
 
async def sequential():
    start = time.perf_counter()
    a = await fetch("a", 2)
    b = await fetch("b", 2)
    c = await fetch("c", 2)
    print(f"sequential: {time.perf_counter() - start:.1f}s")
Python
 
a: start
a: done
b: start
b: done
c: start
c: done
sequential: 6.0s


await fetch("b", 2) doesn't begin until a is finished. While a waits, the loop has nothing else to run, because the function hasn't handed it anything else. Three two-second waits end to end is six seconds. This is the same as the for loop from the introduction, but just written with async.

Now the same three fetches with create_task():

Python
 
async def concurrent():
    start = time.perf_counter()
    task_a = asyncio.create_task(fetch("a", 2))
    task_b = asyncio.create_task(fetch("b", 2))
    task_c = asyncio.create_task(fetch("c", 2))
    a = await task_a
    b = await task_b
    c = await task_c
    print(f"concurrent: {time.perf_counter() - start:.1f}s")
Python
 
a: start
b: start
c: start
a: done
b: done
c: done
concurrent: 2.0s


All three fetches start before any of them finishes, and the whole thing takes two seconds — the length of the longest wait, not the sum of all of them.

create_task() hands a coroutine to the event loop to run on its own and gives you back a Task immediately without waiting. The three tasks begin running the moment this function reaches its first await, because that's when it hands control back to the event loop. By the time await task_a returns, b and c have been running alongside it the whole time.

Based on the browser tab analogy, a coroutine is a link you haven't clicked yet. await fetch(...) is clicking it and staring at the page until it finishes loading. create_task(fetch(...)) is opening a new tab and moving on, with the event loop flipping between tabs as each one needs attention. The Task is the tab itself. You can come back and wait on it, check whether it's finished, or close it with .cancel().

The same two shapes, with the scraper's real building block:

Python
 
# Sequential — each URL waits for the previous one to fully finish
async def sequential(urls):
    results = []
    for url in urls:
        results.append(await _scrape_and_extract(url))
    return results

# Concurrent — every URL is in flight at once
async def concurrent(urls):
    tasks = [asyncio.create_task(_scrape_and_extract(url)) for url in urls]
    return [await t for t in tasks]


The first is the fifty-minute scraper from the introduction with async keywords added, and it's no faster. The keywords alone don't buy you anything. It is the creation of tasks that does. 

What's Next

We can now turn a sequential I/O-bound workflow into concurrent work. Creating multiple tasks is only the beginning; once several tasks are in flight, we need ways to coordinate them and decide when to collect their results. In the next article, we'll compare asyncio.gather(), asyncio.wait(), and asyncio.as_completed() using the same scraper.

If you'd like to see the differences yourself, the companion repo async-scraper-tutorial lets you run the same request through all five strategies by changing one field. That's a better way to internalize the trade-offs than reading about them, and the same shapes carry over to whatever I/O-bound project you're building next.

Event loop HTML IT career Python (language) Task (computing)

Opinions expressed by DZone contributors are their own.

Related

  • Python Async/Sync: Advanced Blocking Detection and Best Practices (Part 2)
  • Python Async/Sync: Understanding and Solving Blocking (Part 1)
  • How To Scale Your Python Services
  • What Does Synchronization With Asyncio Look Like

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook