Skip to content

SearchTimeline Unhandled API response code: 404 #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
caterpillar1219 opened this issue Apr 25, 2025 · 38 comments
Open

SearchTimeline Unhandled API response code: 404 #248

caterpillar1219 opened this issue Apr 25, 2025 · 38 comments

Comments

@caterpillar1219
Copy link

Anyone having an issue with SearchTimeline? Unhandled API response code: 404

@mohammadh1380
Copy link

Anyone having an issue with SearchTimeline? Unhandled API response code: 404

yes twitter in header need x-client-transaction-id but it have a heavy rate limit

@caterpillar1219
Copy link
Author

So the code base is no longer functional without major update?

@TUNKIEN
Copy link

TUNKIEN commented Apr 25, 2025

I have this problem too.

@caterpillar1219
Copy link
Author

caterpillar1219 commented Apr 25, 2025

@vladkens Save us please, Master

@Jfoen
Copy link

Jfoen commented Apr 25, 2025

Having the same problem when trying to retrieve replies via api.search method

@mohammadh1380
Copy link

@yeyuchen198 can you help for generate x-client-transaction-id ?

@dev0xx1
Copy link

dev0xx1 commented Apr 26, 2025

same issue here. fix would really help

@Nanansim
Copy link

anyone solved this problem? please help

@TUNKIEN
Copy link

TUNKIEN commented Apr 26, 2025

hope it will be resolved soon

@caterpillar1219
Copy link
Author

Please help

@bruhmoment03
Copy link

@vladkens please help

@BonifacioCalindoro
Copy link

Same here

@BonifacioCalindoro
Copy link

BonifacioCalindoro commented Apr 26, 2025

Okay, i managed to fix this, at least temporarily. I'd start working on a PR but i'd need this other PR from a different repo to be passed.

Basically, the solution would need the "x-client-transaction-id" to be calculated before every request to the GQL endpoint, using the method and the path as arguments for the function generate_x_client_transaction_id. My solution implies modifying the queue_client.QueueClient.req method to calculate this x-client-transaction-id header and updating the cookies it receives when doing so, for every request. Would look something like this:

twscrape.queue_client

from twscrape.utils import generate_x_client_transaction_id
# ...rest of the imports...
# ....existing code...
# inside async def req, of QueueClient class:
while True:
            ctx = await self._get_ctx()  # not need to close client, class implements __aexit__
            if ctx is None:
                return None

            try:
                try:
                    ctx.clt.headers["x-client-transaction-id"], new_cookies = await generate_x_client_transaction_id(method, url, ctx.clt)
                    ctx.clt.cookies.update(new_cookies)
                except Exception as e:
                    print('error generating x-client-transaction-id', e)
                rep = await ctx.clt.request(method, url, params=params)
                setattr(rep, "__username", ctx.acc.username)
                await self._check_rep(rep)
# ....rest of the code...

And the async def generate_x_client_transaction_id(method, url, ctx.clt) looks something like this:

twscrape.utils

from x_client_transaction.utils import handle_x_migration_async
from x_client_transaction.transaction import AsyncClientTransaction
# ...rest of the imports....

# ...rest of the code...
async def generate_x_client_transaction_id(method: str, url: str) -> str:
    client = httpx.AsyncClient()
    client.headers.update({
        "Authority": "x.com",
        "Accept-Language": "en-US,en;q=0.9",
        "Cache-Control": "no-cache",
        "Referer": "https://x.com",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
        "X-Twitter-Active-User": "yes",
        "X-Twitter-Client-Language": "en"
    })

    response = await handle_x_migration_async(client)
    method = method.upper()
    path = url.split("x.com")[1]
    ct = AsyncClientTransaction()
    await ct.get_transaction_id(response)
    transaction_id = ct.generate_transaction_id(method=method, path=path)
    return transaction_id, client.cookies

Right now the repo i used to generate the x-client-transaction-id only accepts synchronous methods (requests library), hence the PR i submitted to the repo, so hopefully the mantainer passes it soon so we can add its pip package as a dependency for this repo and fix this x-client-transaction-id thing. If you can't wait you can get my fork of his repo and implement the solution into your thing

Extra comment: I didn't need to pass the x-client-transaction-id when loggin in to the account, but sometimes the login failed. Just deleted the account, added it again, and logged in again until successful login

@Rachetage2
Copy link

Rachetage2 commented Apr 26, 2025

Okay, i managed to fix this, at least temporarily. I'd start working on a PR but i'd need this other PR from a different repo to be passed.

Basically, the solution would need the "x-client-transaction-id" to be calculated before every request to the GQL endpoint, using the method and the path as arguments for the function generate_x_client_transaction_id. My solution implies modifying the queue_client.QueueClient.req method to calculate this x-client-transaction-id header and updating the cookies it receives when doing so, for every request. Would look something like this:

from twscrape.utils import generate_x_client_transaction_id

...rest of the imports...

....existing code...

while True:
ctx = await self._get_ctx() # not need to close client, class implements aexit
if ctx is None:
return None

        try:
            try:
                ctx.clt.headers["x-client-transaction-id"], new_cookies = await generate_x_client_transaction_id(method, url, ctx.clt)
                ctx.clt.cookies.update(new_cookies)
            except Exception as e:
                print('error generating x-client-transaction-id', e)
            rep = await ctx.clt.request(method, url, params=params)
            setattr(rep, "__username", ctx.acc.username)
            await self._check_rep(rep)

....rest of the code...

And the async def generate_x_client_transaction_id(method, url, ctx.clt) looks something like this:

async def generate_x_client_transaction_id(method: str, url: str) -> str:
client = httpx.AsyncClient()
client.headers.update({
"Authority": "x.com",
"Accept-Language": "en-US,en;q=0.9",
"Cache-Control": "no-cache",
"Referer": "https://x.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"X-Twitter-Active-User": "yes",
"X-Twitter-Client-Language": "en"
})

response = await handle_x_migration_async(client)
method = method.upper()
path = url.split("x.com")[1]
ct = AsyncClientTransaction()
await ct.get_transaction_id(response)
transaction_id = ct.generate_transaction_id(method=method, path=path)
return transaction_id, client.cookies

Right now the repo i used to generate the x-client-transaction-id only accepts synchronous methods (requests library), hence the PR i submitted to the repo, so hopefully the mantainer passes it soon so we can add its pip package as a dependency for this repo and fix this x-client-transaction-id thing. If you can't wait you can get my fork of his repo and implement the solution into your thing

Extra comment: I didn't need to pass the x-client-transaction-id when loggin in to the account, but sometimes the login failed. Just deleted the account, added it again, and logged in again until successful login

Hey thank you very much, question, how to fully create generate_x_client_transaction_id? I see that in the function you use other functions, but they are not in twscrape code (handle_x_migration_async), from where do we import them?

Thank you!

EDIT: Just found it in your fork, thank brother, God bless you. Hope we'll have a fix soon in Twscrape.

@BonifacioCalindoro
Copy link

Okay, i managed to fix this, at least temporarily. I'd start working on a PR but i'd need this other PR from a different repo to be passed.
Basically, the solution would need the "x-client-transaction-id" to be calculated before every request to the GQL endpoint, using the method and the path as arguments for the function generate_x_client_transaction_id. My solution implies modifying the queue_client.QueueClient.req method to calculate this x-client-transaction-id header and updating the cookies it receives when doing so, for every request. Would look something like this:
from twscrape.utils import generate_x_client_transaction_id

...rest of the imports...

....existing code...

while True:
ctx = await self._get_ctx() # not need to close client, class implements aexit
if ctx is None:
return None

        try:
            try:
                ctx.clt.headers["x-client-transaction-id"], new_cookies = await generate_x_client_transaction_id(method, url, ctx.clt)
                ctx.clt.cookies.update(new_cookies)
            except Exception as e:
                print('error generating x-client-transaction-id', e)
            rep = await ctx.clt.request(method, url, params=params)
            setattr(rep, "__username", ctx.acc.username)
            await self._check_rep(rep)

....rest of the code...

And the async def generate_x_client_transaction_id(method, url, ctx.clt) looks something like this:
async def generate_x_client_transaction_id(method: str, url: str) -> str:
client = httpx.AsyncClient()
client.headers.update({
"Authority": "x.com",
"Accept-Language": "en-US,en;q=0.9",
"Cache-Control": "no-cache",
"Referer": "https://x.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"X-Twitter-Active-User": "yes",
"X-Twitter-Client-Language": "en"
})

response = await handle_x_migration_async(client)
method = method.upper()
path = url.split("x.com")[1]
ct = AsyncClientTransaction()
await ct.get_transaction_id(response)
transaction_id = ct.generate_transaction_id(method=method, path=path)
return transaction_id, client.cookies

Right now the repo i used to generate the x-client-transaction-id only accepts synchronous methods (requests library), hence the PR i submitted to the repo, so hopefully the mantainer passes it soon so we can add its pip package as a dependency for this repo and fix this x-client-transaction-id thing. If you can't wait you can get my fork of his repo and implement the solution into your thing
Extra comment: I didn't need to pass the x-client-transaction-id when loggin in to the account, but sometimes the login failed. Just deleted the account, added it again, and logged in again until successful login

Hey thank you very much, question, how to fully create generate_x_client_transaction_id? I see that in the function you use other functions, but they are not in twscrape code (handle_x_migration_async), from where do we import them?

Thank you!

Check the repo i forked and edited at https://github.com/BonifacioCalindoro/XClientTransaction

There you will find the necessary code to patch your twscrape

Also I updated my original comment with a bit more explanation of the imports and where to place the code👍

@Rachetage2
Copy link

Okay, i managed to fix this, at least temporarily. I'd start working on a PR but i'd need this other PR from a different repo to be passed.
Basically, the solution would need the "x-client-transaction-id" to be calculated before every request to the GQL endpoint, using the method and the path as arguments for the function generate_x_client_transaction_id. My solution implies modifying the queue_client.QueueClient.req method to calculate this x-client-transaction-id header and updating the cookies it receives when doing so, for every request. Would look something like this:
from twscrape.utils import generate_x_client_transaction_id

...rest of the imports...

....existing code...

while True:
ctx = await self._get_ctx() # not need to close client, class implements aexit
if ctx is None:
return None

        try:
            try:
                ctx.clt.headers["x-client-transaction-id"], new_cookies = await generate_x_client_transaction_id(method, url, ctx.clt)
                ctx.clt.cookies.update(new_cookies)
            except Exception as e:
                print('error generating x-client-transaction-id', e)
            rep = await ctx.clt.request(method, url, params=params)
            setattr(rep, "__username", ctx.acc.username)
            await self._check_rep(rep)

....rest of the code...

And the async def generate_x_client_transaction_id(method, url, ctx.clt) looks something like this:
async def generate_x_client_transaction_id(method: str, url: str) -> str:
client = httpx.AsyncClient()
client.headers.update({
"Authority": "x.com",
"Accept-Language": "en-US,en;q=0.9",
"Cache-Control": "no-cache",
"Referer": "https://x.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"X-Twitter-Active-User": "yes",
"X-Twitter-Client-Language": "en"
})

response = await handle_x_migration_async(client)
method = method.upper()
path = url.split("x.com")[1]
ct = AsyncClientTransaction()
await ct.get_transaction_id(response)
transaction_id = ct.generate_transaction_id(method=method, path=path)
return transaction_id, client.cookies

Right now the repo i used to generate the x-client-transaction-id only accepts synchronous methods (requests library), hence the PR i submitted to the repo, so hopefully the mantainer passes it soon so we can add its pip package as a dependency for this repo and fix this x-client-transaction-id thing. If you can't wait you can get my fork of his repo and implement the solution into your thing
Extra comment: I didn't need to pass the x-client-transaction-id when loggin in to the account, but sometimes the login failed. Just deleted the account, added it again, and logged in again until successful login

Hey thank you very much, question, how to fully create generate_x_client_transaction_id? I see that in the function you use other functions, but they are not in twscrape code (handle_x_migration_async), from where do we import them?
Thank you!

Check the repo i forked and edited at https://github.com/BonifacioCalindoro/XClientTransaction

There you will find the necessary code to patch your twscrape

Also I updated my original comment with a bit more explanation of the imports and where to place the code👍

Thanks a lot, successfully patched!

@Rachetage2
Copy link

Btw sometimes it still 404: 2025-04-27 01:45:16.955 | ERROR | twscrape.queue_client:_check_rep:199 - Unhandled API response code: 404 - 41/50 - XXXXXXXXX- OK

@TUNKIEN
Copy link

TUNKIEN commented Apr 27, 2025

2025-04-27 07:42:47.367 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:301 - No account available for queue "SearchTimeline". Next available at 07:43:25
2025-04-27 07:43:27.436 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:308 - Continuing with account twitter_scraper on queue SearchTimeline
2025-04-27 07:43:27.922 | ERROR | twscrape.queue_client:_check_rep:197 - Unhandled API response code: 520 - -1/-1 - twitter_scraper - OK
2025-04-27 07:43:27.929 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:301 - No account available for queue "SearchTimeline". Next available at 07:54:43

I still have this error

@BonifacioCalindoro
Copy link

Btw sometimes it still 404: 2025-04-27 01:45:16.955 | ERROR | twscrape.queue_client:_check_rep:199 - Unhandled API response code: 404 - 41/50 - XXXXXXXXX- OK

Yeah, also happens to me, it's strange because it hits the 404 unexpectedly, sometimes it does sometimes it doesn't.

It's not the most graceful handling, but I also hardcoded the account lock time in that scenario to just 5 seconds instead of 15 min

@BonifacioCalindoro
Copy link

2025-04-27 07:42:47.367 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:301 - No account available for queue "SearchTimeline". Next available at 07:43:25
2025-04-27 07:43:27.436 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:308 - Continuing with account twitter_scraper on queue SearchTimeline
2025-04-27 07:43:27.922 | ERROR | twscrape.queue_client:_check_rep:197 - Unhandled API response code: 520 - -1/-1 - twitter_scraper - OK
2025-04-27 07:43:27.929 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:301 - No account available for queue "SearchTimeline". Next available at 07:54:43

I still have this error

Apparently the error 520 has to do with cloudflare not being able to correctly process the server's response, so I don't know if it has to do with this specific library. I never got that 520 error before

@Rachetage2
Copy link

Btw sometimes it still 404: 2025-04-27 01:45:16.955 | ERROR | twscrape.queue_client:_check_rep:199 - Unhandled API response code: 404 - 41/50 - XXXXXXXXX- OK

Yeah, also happens to me, it's strange because it hits the 404 unexpectedly, sometimes it does sometimes it doesn't.

It's not the most graceful handling, but I also hardcoded the account lock time in that scenario to just 5 seconds instead of 15 min

Yep, just deactivated the lock too

I'm investigating, will let you know if I find something

@Rachetage2
Copy link

Rachetage2 commented Apr 27, 2025

Btw sometimes it still 404: 2025-04-27 01:45:16.955 | ERROR | twscrape.queue_client:_check_rep:199 - Unhandled API response code: 404 - 41/50 - XXXXXXXXX- OK

Yeah, also happens to me, it's strange because it hits the 404 unexpectedly, sometimes it does sometimes it doesn't.

It's not the most graceful handling, but I also hardcoded the account lock time in that scenario to just 5 seconds instead of 15 min

I think the problem comes from the fact that with each request we refresh the page, instead of loading the home page once per account (this is the normal behavior of an X user)

If you go to X in your browser, you'll sometimes see that you also have a 404 bug, you have to refresh for your session to work

We'd need the same thing to solve the problem, one session per account + load the home page until it works, then we can generate our transaction ids without any problem

Why do other X scrapers work? Because they don't have a pool system like we do, so they load the home page once, and then there's no problem (if it's successful).

@vladkens Master if you have some time, we'd need your help to fix it

@vbresan
Copy link

vbresan commented Apr 27, 2025

Why do other X scrapers work? Because they don't have a pool system like we do, so they load the home page once, and then there's no problem (if it's successful).

@Rachetage2 Which other X scrapers?

@Rachetage2
Copy link

Why do other X scrapers work? Because they don't have a pool system like we do, so they load the home page once, and then there's no problem (if it's successful).

@Rachetage2 Which other X scrapers?

Like twikit, but twscrape >> with its pool system

@tadasgedgaudas
Copy link

twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141

Could be brought here who ever is working on it

@BonifacioCalindoro
Copy link

twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141

Could be brought here who ever is working on it

Yeah they basically implemented what that guy on the repo i posted did but async, could be easily ported in this repo actually...

@Rachetage2
Copy link

twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141
Could be brought here who ever is working on it

Yeah they basically implemented what that guy on the repo i posted did but async, could be easily ported in this repo actually...

Any info about the 404 that often appears?

@BonifacioCalindoro
Copy link

twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141
Could be brought here who ever is working on it

Yeah they basically implemented what that guy on the repo i posted did but async, could be easily ported in this repo actually...

Any info about the 404 that often appears?

not working on it atm as i already "stabilized" my service with the temporary patch i suggested yesterday, will look more into it when i got time

@Rachetage2
Copy link

twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141
Could be brought here who ever is working on it

Yeah they basically implemented what that guy on the repo i posted did but async, could be easily ported in this repo actually...

Any info about the 404 that often appears?

not working on it atm as i already "stabilized" my service with the temporary patch i suggested yesterday, will look more into it when i got time

Ok man, thanks

One solution would be to only load the homepage one time, if the info already made one valid tid, it only ba valid ones after, so generate, if 404, generate again, if it works, save home page response to reuse it

@vladkens
Copy link
Owner

Hi everyone, I see this error with x-client-transaction-id. I will try to make a release with a fix within a couple of days.

Thanks to @BonifacioCalindoro for investigating the problem and @fa0311 for reverse engenering the key generation algorithm.


References:

@Rachetage2
Copy link

Hi everyone, I see this error with x-client-transaction-id. I will try to make a release with a fix within a couple of days.

Thanks to @BonifacioCalindoro for investigating the problem and @fa0311 for reverse engenering the key generation algorithm.

References:

Love you man

@mika-jpd
Copy link
Contributor

mika-jpd commented Apr 28, 2025

@vladkens if you're interested, I've been working on making the following library's implementation of the reverse-engineered process described in the https://antibot.blog/posts/ website work with twscrape.

The implementation is here:
https://github.com/iSarabjitDhiman/TweeterPy/tree/master/tweeterpy/tid

EDIT - just read the thread above and it makes reference to this repo. Sorry for redundant comment !

vladkens added a commit that referenced this issue Apr 29, 2025
@vladkens
Copy link
Owner

I added this very strange code to generate x-client-transaction-id. This approach doesn't look stable enough for me right now (404 is returned quite often and the code has to be re-requested). Released in v0.17.0. Let's see how it works. If anything, write in this issue

pip install --upgrade twscrape

@vladkens vladkens pinned this issue Apr 29, 2025
@JAAVAAD
Copy link

JAAVAAD commented Apr 29, 2025

pip install --upgrade twscrape

Hi @vladkens , thanks
I tested it but it doesn't have the required persistence code and after crawling a few posts, it won't crawl anymore.

@caterpillar1219
Copy link
Author

Can we have twikit implementation ported to twscrape? They don't have the 404 issue somehow...

@Rachetage2
Copy link

I'm not sure how you did it, but If we scrape home page every request, that's why it's not stable.
It seems only one scrape is enough per account session, to then generate all transaction ids, by passing the home page response again and again.

@Rachetage2
Copy link

I added this very strange code to generate x-client-transaction-id. This approach doesn't look stable enough for me right now (404 is returned quite often and the code has to be re-requested). Released in v0.17.0. Let's see how it works. If anything, write in this issue

pip install --upgrade twscrape

On my side it's pretty stable for the moment (not using limit btw), thanks for the fix, it's highly appreciated, won't forgive to support you after launching my project

@Rachetage2
Copy link

It seems the 404 problem occured because of problems in tid calcul, they resolved it in the new version.

See: iSarabjitDhiman/XClientTransaction#10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

15 participants