Python Asyncio: Understanding Concurrency With a Scraper
Use asyncio.create_task() for I/O concurrency, gather or wait to manage task batches, and a Semaphore to prevent resource exhaustion.
Introduction
If you need to scrape a few thousand job postings from a careers page, one way to do it is by using a for loop, one request after another:
import requests
def scrape_all(urls):
pages = []
for url in urls:
response = requests.get(url) # blocks here until the page comes back
pages.append(response.text)
return pages
(The real scraper uses a headless browser rather than requests, but the shape is identical.)
This works, but it's slow. For example, if a page takes two seconds to come back, then 1500 pages will take 3000 seconds (50 minutes). Most of that time is spent not computing anything. The program is sitting around waiting for responses to cross the network. While it waits on page 1, it can't even ask for page 2.
Python has more than one answer to "how do I stop wasting that wait?". The first is concurrency, while the second is parallelism.
Concurrency vs Parallelism
The analogy that explains the difference is when you have two YouTube videos open in two browser tabs, and both are buffering.
- Concurrency is watching tab 1, having it stall to buffer, switching to tab 2 while tab 1 catches up, then switching back once tab 1 is ready. You're still only watching one video at any given instant, but nothing sits idle for long, and the moment one thing is blocked, your attention moves to whatever isn't.
- Parallelism is watching both videos at the same time, side by side, with two separate screens and two separate sets of eyes. No switching and no waiting on one for the other. They're happening simultaneously.
Concurrency is about overlapping waiting, and it works even on a single CPU core. Most of that "watching" time is actually idle time. Parallelism is about overlapping working, and it requires multiple CPU cores doing computation at the same moment. You can't parallelize your way out of a slow network connection, but you should be able to get better at not wasting the wait.
That maps directly onto the two categories of workload:
- CPU-bound work (image processing, number crunching, data transformation) is limited by how fast the processor can compute. This is where parallelism helps since what you need are more cores, more simultaneous computation. Python's tool for this is the
multiprocessingmodule. It sidesteps the GIL by running separate processes with separate interpreters. - I/O-bound work (network requests, waiting on an LLM API, database queries) is limited by how long you sit around waiting for something external to respond. This is where concurrency helps because while one request is "buffering," you can be making progress on another. There are two different tools for working with concurrency. The first is called
threading, while the second is calledasyncio.
Why Async Fits I/O
This article is about asyncio. It uses cooperative, single-threaded concurrency. In this case, your code explicitly says "I'm waiting, someone else can go now" via await, instead of the OS pre-emptively switching between threads for you. For a scraper, asyncio is a better fit because you are short on patience while 1,500 HTTP requests are all "buffering" at once.
asyncio lets a single thread juggle thousands of these waiting operations concurrently. While one task waits for a response from OpenAI, another can wait for a Playwright page load. Neither is burning CPU time while it waits, since it's just parked, and the event loop wakes up the thread the moment its data arrives. Nothing is happening in parallel because there's still only one thread, but most of the waiting is not wasted.
This article explains asyncio's concurrency primitives by building a real scraper. Given a company's careers page, it:
- Loads the page and collects the URL of every job posting. For each of those URLs, loads the posting and asks OpenAI to pull out structured fields.
Step 2 is where all the waiting happens. The reason is that it's the same two slow operations repeated for every URL. These two are the only pieces of the scraper you need to know about:
async def scrape_webpage(url: str) -> list[str]:
"""Loads a page in a headless browser (Playwright) and returns its rendered HTML."""
async def extract_job_information(html_document: str, apply_url: str) -> dict:
"""Sends the page text to OpenAI and returns structured fields."""
A call to the second one returns a dictshaped like this:
{
"job_title": "Senior Data Engineer",
"company_name": "Example Corp",
"location": "Remote - US",
"commitment": "full-time",
"seniority_level": "Senior Level",
"job_description": "We're looking for ...",
"apply_url": "https://example.wd5.myworkdayjobs.com/...",
}
This article explains how to run 1,500 job URLs effectively.
async def process(urls: list[str]) -> list[dict]:
... # scrape and extract every URL, in whatever way this strategy chooses
The Basics
There are three concepts to understand. There are coroutines (async def and await), the event loop, and the difference between await and asyncio.create_task(). The examples below use asyncio.sleep() to stand in for a network request. They run as-is. You can paste them into a file and try them. After each one, we'll connect it to a real scraper.
Coroutines: async def and await
import asyncio
async def fetch(name, seconds):
print(f"{name}: start")
await asyncio.sleep(seconds) # stands in for a network request
print(f"{name}: done")
return name
Three rules cover almost everything you'll run into:
1. async def defines a coroutine function. It looks like a normal function, but it's allowed to pause partway through.
2. Calling it does not run it. You get back a coroutine object.
print(fetch("a", 1))
# <coroutine object fetch at 0x7f8c358a39f0>
# RuntimeWarning: coroutine 'fetch' was never awaited
You can see that a: start never printed. The warning "was never awaited" means you called the function and forgot to await it, so nothing ran.
3. await runs a coroutine and gives you its result. It can only be used inside another async def. asyncio.run()starts the whole chain from regular code.
asyncio.run(fetch("a", 1))
# a: start
# a: done
await does two jobs at once. First, it runs the coroutine and hands back the return value, like a normal function call. Second, it marks a point where this function is willing to pause: "I'm waiting on something slow, so someone else can run while I wait". That second job is the entire reason asyncio exists.
The scraper is built from exactly this. The two awaits are the pause points. The first awaitloads the page while the second await calls the API.
async def _scrape_and_extract(url: str) -> dict:
documents = await scrape_webpage(url)
html = remove_html_tags(documents[0])
return await extract_job_information(html_document=html, apply_url=url)
The scrape_webpage returns a list because it can also follow pagination, and a single job posting is one page; hence documents[0]. And remove_html_tags is an ordinary, non-async helper that strips the markup so the model sees plain text. It has nothing to do with concurrency, so you can ignore it from here on.
The Event Loop: Who Decides What Runs
When a function pauses at an await, "someone else can run" needs an event loop. From the browser tabs analogy earlier, the event loop is you. Each coroutine is a tab. An await is the moment a tab starts buffering, and the event loop is what notices and flips to a tab that's ready.
The event loop keeps track of which tasks are ready to run and which are waiting on something (a response, a timer). It runs one ready task until that task hits an await on something that isn't finished yet, sets it aside, and picks the next ready one. When whatever it was waiting for completes, the task goes back on the ready list. All of this happens on a single thread. Only one piece of your code is running at any instant. Concurrency comes from the waiting overlapping, not from the code running simultaneously.
There are two things to consider:
- Switching only happens at an
await. Nothing interrupts a task midway. So if a coroutine calls something slow without awaiting it (e.g.,time.sleep(2)orrequests.get(url)), the loop can't switch away, and every other task pauses until that call returns. Inside async code you useawait asyncio.sleep(2)and async APIs (Playwright's async API, LangChain'sainvoke) instead of their blocking equivalents. This is why the scraper usesasync_playwrightrather than a plain HTTP library. - You rarely create the loop yourself.
asyncio.run()creates one for a script. In a FastAPI app like this one, uvicorn creates it and runs every request handler as a task on it.
await vs. asyncio.create_task(): Sequential vs Concurrent
It is important to understand that writing await does not make anything concurrent by itself.
async def sequential():
start = time.perf_counter()
a = await fetch("a", 2)
b = await fetch("b", 2)
c = await fetch("c", 2)
print(f"sequential: {time.perf_counter() - start:.1f}s")
a: start
a: done
b: start
b: done
c: start
c: done
sequential: 6.0s
await fetch("b", 2) doesn't begin until a is finished. While a waits, the loop has nothing else to run, because the function hasn't handed it anything else. Three two-second waits end to end is six seconds. This is the same as the for loop from the introduction, but just written with async.
Now the same three fetches with create_task():
async def concurrent():
start = time.perf_counter()
task_a = asyncio.create_task(fetch("a", 2))
task_b = asyncio.create_task(fetch("b", 2))
task_c = asyncio.create_task(fetch("c", 2))
a = await task_a
b = await task_b
c = await task_c
print(f"concurrent: {time.perf_counter() - start:.1f}s")
a: start
b: start
c: start
a: done
b: done
c: done
concurrent: 2.0s
All three fetches start before any of them finishes, and the whole thing takes two seconds — the length of the longest wait, not the sum of all of them.
create_task() hands a coroutine to the event loop to run on its own and gives you back a Task immediately without waiting. The three tasks begin running the moment this function reaches its first await, because that's when it hands control back to the event loop. By the time await task_a returns, b and c have been running alongside it the whole time.
Based on the browser tab analogy, a coroutine is a link you haven't clicked yet. await fetch(...) is clicking it and staring at the page until it finishes loading. create_task(fetch(...)) is opening a new tab and moving on, with the event loop flipping between tabs as each one needs attention. The Task is the tab itself. You can come back and wait on it, check whether it's finished, or close it with .cancel().
The same two shapes, with the scraper's real building block:
# Sequential — each URL waits for the previous one to fully finish
async def sequential(urls):
results = []
for url in urls:
results.append(await _scrape_and_extract(url))
return results
# Concurrent — every URL is in flight at once
async def concurrent(urls):
tasks = [asyncio.create_task(_scrape_and_extract(url)) for url in urls]
return [await t for t in tasks]
The first is the fifty-minute scraper from the introduction with async keywords added, and it's no faster. The keywords alone don't buy you anything. It is the creation of tasks that does.
What's Next
We can now turn a sequential I/O-bound workflow into concurrent work. Creating multiple tasks is only the beginning; once several tasks are in flight, we need ways to coordinate them and decide when to collect their results. In the next article, we'll compare asyncio.gather(), asyncio.wait(), and asyncio.as_completed() using the same scraper.
If you'd like to see the differences yourself, the companion repo async-scraper-tutorial lets you run the same request through all five strategies by changing one field. That's a better way to internalize the trade-offs than reading about them, and the same shapes carry over to whatever I/O-bound project you're building next.
Opinions expressed by DZone contributors are their own.