Subinterpreters and Asyncio (blog.changs.co.uk)

Python 3.14 late-added PEP-734 subinterpreters to the stdlib under concurrent.interpreters, giving Python a built-in way to run multiple independent interpreters in the same process. That matters because subinterpreters can outperform traditional threading for CPU-bound work while avoiding some of the costs and isolation gaps of multiprocessing. The stdlib exposes two high-level paths: InterpreterPoolExecutor (works like ProcessPoolExecutor but pickles functions/args/results, which can negate performance gains) and create().call_in_thread (uses shared memory and is faster but awkward and doesn’t reuse interpreters). To make subinterpreters practical with async code, the author released aiointerpreters: an asyncio-friendly Runner that spawns worker interpreters plus a coordinator thread, exchanges tasks/results via queues, and resolves asyncio futures (using loop.call_soon_threadsafe). Key constraints reflect subinterpreter design: functions must be module-level and accept/return only “shareable” types (primitives, tuples, Queue, memoryview); functions are loaded inside interpreters via importlib (module name or file path) to avoid pickling. A crawler demo combines async IO for fetching and subinterpreters for CPU-bound HTML parsing (BeautifulSoup), showing better performance than free-threading on the author’s machine. Practical caveats: pickling overhead for the executor, the need to reuse interpreters to avoid high creation cost, and limited data types for zero-copy/shared execution.