Does crawl4AI already have a function to scrape all related URLs from a root URL?

[QuangTQV]

Description: Starting from a root URL, it will navigate to child URLs, scraping content in parallel while collecting links. It can utilize a queue or some other mechanism and should allow setting a maximum depth parameter. It returns a list of dictionaries, where each element contains information about a URL (link, content, depth, images, markdown, etc.)

[aravindkarnam]

Hi @QuangTQV. We are currently working on a scraper module that takes in a root URL then does a Breadth First traversal until the configured depth is reached. It's currently under review and testing.

If you are interested in trying this out, you can check this PR
https://github.com/aravindkarnam/crawl4ai/tree/scraper-uc

there's a quick start example in
https://github.com/aravindkarnam/crawl4ai/blob/scraper-uc/docs/scraper/scraper_quickstart.py

Please see if this meets your requirements. We are working on some performance enhancements, post that this will be released.

[1933211129]

@QuangTQV We are working on the same thing, but the current solution based on crawl4ai is relatively slow. Could we discuss the implementation ideas more in the future?😀

[unclecode]

@1933211129 @QuangTQV The new version 0.4.3 includes a very strong component for parallel crawling and will serve as the core for the "scraper" branch. I plan to merge the scraper branch that @aravindkarnam worked on. In the meantime, check this link to gather some ideas for parallel crawling: https://docs.crawl4ai.com/advanced/multi-url-crawling/

[Rm1n90]

@unclecode @aravindkarnam, Is there any update regarding the feature "scrape all related URLs from a root URL"?

[aravindkarnam]

It's almost done @Rm1n90 ! Currently undergoing testing. It will most likely be released within the next week.

[opusmagnum]

Thank you @aravindkarnam! I have tested it on your feature-branch and it works great.

[Dev4011]

Thanks @aravindkarnam, this is really very useful - awaiting it's release.

[flowluap]

Will this be implemented by the api?

I think this is very useful also for non python users...

[aravindkarnam]

@flowluap Yes. It's on the roadmap.

[tezzcan]

@aravindkarnam hello, as far as i can see this feature is not released yet. any updates?? excellent work btw I'm waiting like a fly rubbing its hands together.

[aravindkarnam]

@tezzcan Really appreciate your enthusiasm. This is what keeps us motivated to build and maintain Crawl4AI. The release should happen most likely tonight 🤞🏼. This is one of the major releases in the history of project so far, that's why we are taking some time to prepare everything.

Along with the release there are also a ton of super helpful tutorial videos that are going to come out through the week. So stay tuned on changelog and @unclecode 's x handle for updates.

[1933211129]

Looking forward to it! We are also paying attention to this area of work, thank you for your contributions!