PYTHON TOOLBOX

Aiomultiprocess: Super Easy Integrate Multiprocessing & Asyncio in Python

Even no need to know much about asyncio and multiprocessing

Aiomultiprocess makes your code fast and easy. Photo Credit: Created by Author, Canva

In this article, I will introduce how to integrate multiprocessing and asyncio using the aiomultiprocess library easily. The article includes a web scraping project example and the best practices for using this library.

Introduction

My colleague Wang came to me today and said that his boss assigned him a new task: to write web scraping code to fetch information from 1,000 books on books.toscrape.com as quickly as possible.

Wang told me: “I’ve read your related articles, and since the boss has performance requirements, why don’t I write one using asyncio? It doesn’t seem too difficult.”

“30.09 seconds,” I told him a number.

“What’s that?” Wang asked.

I said I had just tried it and that only using concurrent tasks with asyncio for web scraping would take that long on my computer. This speed is already relatively fast.

“12.64 seconds,” I told him another number.

The speed doubled! Wang was stunned.