Async Python and concurrency
Run many slow I/O calls at once with async/await, and know when threads or processes fit better.
Why it matters: LLM calls wait on the network. Async runs many of them together so a batch finishes in seconds, not minutes.
Basics
What async does
async def makes a coroutine. await pauses it so other work can run.
asyncio.gather(...) runs several coroutines together. That helps I/O (HTTP, DB, LLM APIs), not heavy CPU math.
Common mistake: calling a coroutine without await - nothing runs.
Tiny example
Await two short sleeps at the same time.
import asyncio
import time
async def ping(i):
await asyncio.sleep(0.05)
return i
start = time.perf_counter()
print(await asyncio.gather(ping(1), ping(2)))
print("sec", round(time.perf_counter() - start, 3))Basics
Bound concurrency
A Semaphore caps how many tasks run at once. Unlimited gather calls can hit rate limits (429).
Do not use blocking time.sleep or sync requests inside async - that freezes the whole event loop. Prefer asyncio.sleep or asyncio.to_thread.
Rule of thumb: asyncio for many API waits; threads for blocking libs; processes for heavy CPU.
Tiny example
Limit how many jobs run at once.
import asyncio
sem = asyncio.Semaphore(2)
async def job(i):
async with sem:
await asyncio.sleep(0.01)
return i
print(await asyncio.gather(*(job(i) for i in range(4))))Then AI / ML
AI / ML examples
Same idea, used in real AI work. Press Try in lab to run it.
AI / ML example
AI / ML: sequential vs concurrent LLM waits
Five fake 0.3s calls: about 1.5s in a row, about 0.3s together.
In AI / ML: LLM and embedding HTTP calls are I/O waits - async cuts wall time.
example.py
Python
import asyncio
import time
async def fake_llm(i):
await asyncio.sleep(0.3)
return f"answer-{i}"
start = time.perf_counter()
seq = [await fake_llm(i) for i in range(5)]
seq_ms = (time.perf_counter() - start) * 1000
start = time.perf_counter()
conc = await asyncio.gather(*(fake_llm(i) for i in range(5)))
conc_ms = (time.perf_counter() - start) * 1000
print("seq", seq, f"{seq_ms:.0f} ms")
print("conc", conc, f"{conc_ms:.0f} ms")AI / ML example 2
AI / ML: keep going when one call fails
return_exceptions=True saves the whole batch.
In AI / ML: Production LLM batches often log failures and keep successes.
example.py
Python
import asyncio
async def maybe_fail(i):
await asyncio.sleep(0.05)
if i % 3 == 0:
raise TimeoutError(f"call {i}")
return f"ok-{i}"
results = await asyncio.gather(
*(maybe_fail(i) for i in range(6)),
return_exceptions=True,
)
ok = [r for r in results if not isinstance(r, Exception)]
bad = [str(r) for r in results if isinstance(r, Exception)]
print("ok:", ok)
print("bad:", bad)Takeaways
- 1async/await helps I/O waits like LLM and HTTP calls.
- 2Never block the event loop with sync sleep or sync HTTP.
- 3Use a Semaphore and return_exceptions for safe batches.