Horizontal scaling - Tasks in the memory

[teehamaral]

Looks like it's memory-oriented for creating new tasks, so how do you make it run in multiple servers horizontally scaling? Because of the way it is now, it will cause inconsistency in querying for the task ID to retrieve the results if the request goes to a server where was not created the task.

Also when creating tasks via /crawl endpoint, including multiple URLs (about 10 URLs), it consumes a good amount of memory, I was able to see peaks of 99%.

Does anyone already have this kind of problem?

[unclecode]

Hi @teehamaral Let's start with the second part of your message. I looked at your messages from last week, so I assume you were using the previous Docker. Try the new one that is available, and I will discuss it more next week because the new version uses entirely new memory management tools and is quite different. We designed it to be extremely efficient in memory usage. This version will undergo testing in Google Cloud Functions, which is not easy at all, but it works great.

Regarding the first point, building a decentralized model is on the roadmap. For the decentralized approach, I am creating an engine from the ground up, which does not typically use the standard Docker. It spawns all the crawl tasks among clusters, with each cluster containing nodes that can collect and combine data before bringing it back. It employs classic algorithms from computer networks, which we will implement in the second quarter. Right now, the most important focus for me has been efficiency and memory usage. So, please try the new Docker and report any issues; we will fix them to ensure it works effectively. Thank you so much.

[teehamaral]

@unclecode Yes, I'm using the previous Docker. I saw that on the new version, the endpoints are different, previously it was starting tasks through /crawl and later through /task/{task-id} endpoint we could get the results of the process. This new approach would not have the issue of scaling since the client will need to wait for the server to respond via API request, correct?

[unclecode]

@teehamaral Yes, you’re right! T h after thinking it through, I realized I don’t need to handle such features like task-queue. Bcoz it’s more relevant to how someone wants to use Crawl4AI in their own product, and not the Crawl4ai concern.

If someone needs a queue, they can implement it themselves. My layer just exposes Crawl4AI’s functionality directly, which is why I removed that part.

The only optional feature I kept is JWT token support, which is OFF by default, but if enabled, the server provides a JWT token as a bonus. However, that’s not part of Crawl4AI itself, just an extra for the server. That’s why it’s disabled by default, so developers can choose how to handle authentication.

And yes, no queue, no polling—it simply returns the result. Just be mindful that for tasks like deep crawling, which can take 2-3 minutes (depending on size), you should adjust endpoint runtime settings accordingly. If you’re using it in a production system, make sure to handle long-running tasks properly.