'charmap' codec can't decode byte 0x81 in position 1980: character maps to <undefined>

[jhuang33]

Hi, I'm trying to web scrape cyber event related sites using crawl4ai and deepseek but I keep encountering this error. I tried following example implementations from the site and videos but they all get the same error any help? Code below

Extracted Items: [{'index': 0, 'error': True, 'tags': ['error'], 'content': "'charmap' codec can't decode byte 0x81 in position 1980: character maps to "}]

import asyncio
import json
import os
from typing import List

from crawl4ai import AsyncWebCrawler, BrowserConfig, CacheMode, CrawlerRunConfig, LLMConfig
from crawl4ai.extraction_strategy import LLMExtractionStrategy
from pydantic import BaseModel, Field

URL_TO_SCRAPE = "https://www.bleepingcomputer.com/news/security/toyota-confirms-third-party-data-breach-impacting-customers/"

INSTRUCTION_TO_LLM = (
"From the source, answer the following with one word and if it can't be determined answer with Undetermined: "
"Threat actor type (Criminal, Hobbyist, Hacktivist, State Sponsored, etc), Industry, "
"Motive (Financial, Political, Protest, Espionage, Sabotage, etc), Event Type (Exploitive, Disruptive, Mixed), Country, State, County. "
)

class ThreatIntel(BaseModel):
threat_actor_type: str = Field(..., alias="Threat actor type")
industry: str
motive: str
event_type: str
country: str
state: str
county: str

async def main():

deepseek_config = LLMConfig(
    provider="deepseek/deepseek-chat",
    api_token="********"
)

llm_strategy = LLMExtractionStrategy(
    llm_config=deepseek_config,
    schema=ThreatIntel.model_json_schema(),
    extraction_type="schema",
    instruction=INSTRUCTION_TO_LLM,
    chunk_token_threshold=1000,
    overlap_rate=0.0,
    apply_chunking=True,
    input_format="markdown",
    extra_args={"temperature": 0.0, "max_tokens": 800},
)

crawl_config = CrawlerRunConfig(
    extraction_strategy=llm_strategy,
    cache_mode=CacheMode.BYPASS,
    process_iframes=False,
    remove_overlay_elements=True,
    exclude_external_links=True,
)

browser_cfg = BrowserConfig(headless=True, verbose=True)

async with AsyncWebCrawler(config=browser_cfg) as crawler:

    result = await crawler.arun(url=URL_TO_SCRAPE, config=crawl_config)

    if result.success:
        data = json.loads(result.extracted_content)

        print("Extracted Items:", data)

        llm_strategy.show_usage()
    else:
        print("Error:", result.error_message)

if name == "main":
asyncio.run(main())

[SECVBulRep]

I have same situation

[BatisteOdonatech]

Same here

[BatisteOdonatech]

Fresh install
crawl4ai-doctor says all good
crwl https://www.nbcnews.com/business -o markdown
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1980: character maps to

[yaroofie]

Me too

[JaphetStone]

Yes same problem here, I managed to isolate the problem to it being somewhere in the data pipeline within LiteLLM. It has something to do with the UTF-8 encoding proces.

I fixed it by downgrading the LiteLLM version so it seems to be a version bug. This gave other bugs so I switched to non-llm scraping but that might help you?