[Feature Request]: Create an official Crawl4AI MCP Server

[vanetreg]

What needs to be done?

Reach Crawl4AI's features via the standardized MCP protocol.

What problem does this solve?

Allows easier integration to LLM based systems

Target users/beneficiaries

developers

Current alternatives/workarounds

No response

Proposed approach

No response

[ritvij14]

@unclecode sir, I saw your comment on another Discussion #540, and your github link for an MCP is broken now.

If someone could make this, it would be amazing! I actually tried making it this weekend but am running into a very weird issue regarding this. When the crawler starts the process using the basic method outlined in the documentation here: https://crawl4ai.com/mkdocs/core/simple-crawling/

The MCP server just stops executing in Cursor AI.

In Claude Desktop I get a weird error message.

Claude Desktop Error

{"context":"connection","stack":"SyntaxError: Unexpected token 'I', \"[INIT].... \"... is not valid JSON\n    at JSON.parse (<anonymous>)\n    at I0e (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:189)\n    at R0e.readMessage (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:115)\n    at k0e.processReadBuffer (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1842)\n    at Socket.<anonymous> (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1523)\n    at Socket.emit (node:events:518:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at Pipe.onStreamRead (node:internal/stream_base_commons:191:23)"}
2025-02-23T15:43:03.899Z [crawler] [error] Unexpected token 'F', "[FETCH]... "... is not valid JSON {"context":"connection","stack":"SyntaxError: Unexpected token 'F', \"[FETCH]... \"... is not valid JSON\n    at JSON.parse (<anonymous>)\n    at I0e (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:189)\n    at R0e.readMessage (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:115)\n    at k0e.processReadBuffer (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1842)\n    at Socket.<anonymous> (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1523)\n    at Socket.emit (node:events:518:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at Pipe.onStreamRead (node:internal/stream_base_commons:191:23)"}
2025-02-23T15:43:03.899Z [crawler] [error] Unexpected token 'C', "[COMPLETE] "... is not valid JSON {"context":"connection","stack":"SyntaxError: Unexpected token 'C', \"[COMPLETE] \"... is not valid JSON\n    at JSON.parse (<anonymous>)\n    at I0e (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:189)\n    at R0e.readMessage (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:56:115)\n    at k0e.processReadBuffer (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1842)\n    at Socket.<anonymous> (/Applications/Claude.app/Contents/Resources/app.asar/.vite/build/index.js:57:1523)\n    at Socket.emit (node:events:518:28)\n    at addChunk (node:internal/streams/readable:561:12)\n    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)\n    at Readable.push (node:internal/streams/readable:392:5)\n    at Pipe.onStreamRead (node:internal/stream_base_commons:191:23)"}

this log is directly from Claude Desktop logs. After this, I actually do get the correct result for the crawler's result. But I really need the MCP server to work in Cursor as well.

According to o3-mini a possible reason for this could be:
When FastMCP runs your tool, it manages its own asyncio event loop. The Crawl4AI methods (crawler.start() and crawler.close()) may be expecting to run in a different (or more isolated) async context. In other words, explicitly starting and closing the crawler could be creating conflicts or deadlocks in the already-running event loop, causing the tool call to never finish or to fail.

@unclecode sir, any idea on what this error is, and also if you could share if this is already in the roadmap for Crawl4AI ?

[aravindkarnam]

@ritvij14 Thanks for taking interset. We are considering it and I just started planning for MCP server.

Once ready I'll list down our plan here and open it for comments and feedback.

[GobinFan]

How long will it take for the MCP server to be released? I am looking forward to it. @aravindkarnam

[aravindkarnam]

@GobinFan Currently we are focusing on the docker setup and testing. We'll take this up later. I'll update this thread, when it's picked up for execution.

[ritvij14]

@aravindkarnam I love crawl4AI's abilities and would like to give it a shot, if possible. I have already built a very basic MCP server using Crawl4AI's python package, that scrapes the URL the user has asked. It works great in both Cursor AI and Claude Desktop. I will go through the docs and expand it according to the complete set of functionalities. Is there anyway I can contribute my code to this repository or some way to build that into the official MCP server for Crawl4AI?

[aravindkarnam]

@ritvij14 Awesome! 💯. We are indeed looking for contributions in this area.

I really appreciate the effort. Go ahead and set up a repository with a readme file and share the link here. I'll go through it and come back to you with any questions/feedback. Excited to collaborate with you on this.