-
-
Notifications
You must be signed in to change notification settings - Fork 201
SearchTimeline Unhandled API response code: 404 #248
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
yes twitter in header need x-client-transaction-id but it have a heavy rate limit |
So the code base is no longer functional without major update? |
I have this problem too. |
@vladkens Save us please, Master |
Having the same problem when trying to retrieve replies via api.search method |
@yeyuchen198 can you help for generate x-client-transaction-id ? |
same issue here. fix would really help |
anyone solved this problem? please help |
hope it will be resolved soon |
Please help |
@vladkens please help |
Same here |
Okay, i managed to fix this, at least temporarily. I'd start working on a PR but i'd need this other PR from a different repo to be passed. Basically, the solution would need the "x-client-transaction-id" to be calculated before every request to the GQL endpoint, using the method and the path as arguments for the function generate_x_client_transaction_id. My solution implies modifying the queue_client.QueueClient.req method to calculate this x-client-transaction-id header and updating the cookies it receives when doing so, for every request. Would look something like this: twscrape.queue_clientfrom twscrape.utils import generate_x_client_transaction_id
# ...rest of the imports...
# ....existing code...
# inside async def req, of QueueClient class:
while True:
ctx = await self._get_ctx() # not need to close client, class implements __aexit__
if ctx is None:
return None
try:
try:
ctx.clt.headers["x-client-transaction-id"], new_cookies = await generate_x_client_transaction_id(method, url, ctx.clt)
ctx.clt.cookies.update(new_cookies)
except Exception as e:
print('error generating x-client-transaction-id', e)
rep = await ctx.clt.request(method, url, params=params)
setattr(rep, "__username", ctx.acc.username)
await self._check_rep(rep)
# ....rest of the code... And the async def generate_x_client_transaction_id(method, url, ctx.clt) looks something like this: twscrape.utilsfrom x_client_transaction.utils import handle_x_migration_async
from x_client_transaction.transaction import AsyncClientTransaction
# ...rest of the imports....
# ...rest of the code...
async def generate_x_client_transaction_id(method: str, url: str) -> str:
client = httpx.AsyncClient()
client.headers.update({
"Authority": "x.com",
"Accept-Language": "en-US,en;q=0.9",
"Cache-Control": "no-cache",
"Referer": "https://x.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
"X-Twitter-Active-User": "yes",
"X-Twitter-Client-Language": "en"
})
response = await handle_x_migration_async(client)
method = method.upper()
path = url.split("x.com")[1]
ct = AsyncClientTransaction()
await ct.get_transaction_id(response)
transaction_id = ct.generate_transaction_id(method=method, path=path)
return transaction_id, client.cookies Right now the repo i used to generate the x-client-transaction-id only accepts synchronous methods (requests library), hence the PR i submitted to the repo, so hopefully the mantainer passes it soon so we can add its pip package as a dependency for this repo and fix this x-client-transaction-id thing. If you can't wait you can get my fork of his repo and implement the solution into your thing Extra comment: I didn't need to pass the x-client-transaction-id when loggin in to the account, but sometimes the login failed. Just deleted the account, added it again, and logged in again until successful login |
Hey thank you very much, question, how to fully create generate_x_client_transaction_id? I see that in the function you use other functions, but they are not in twscrape code (handle_x_migration_async), from where do we import them? Thank you! EDIT: Just found it in your fork, thank brother, God bless you. Hope we'll have a fix soon in Twscrape. |
Check the repo i forked and edited at https://github.com/BonifacioCalindoro/XClientTransaction There you will find the necessary code to patch your twscrape Also I updated my original comment with a bit more explanation of the imports and where to place the code👍 |
Thanks a lot, successfully patched! |
Btw sometimes it still 404: 2025-04-27 01:45:16.955 | ERROR | twscrape.queue_client:_check_rep:199 - Unhandled API response code: 404 - 41/50 - XXXXXXXXX- OK |
2025-04-27 07:42:47.367 | INFO | twscrape.accounts_pool:get_for_queue_or_wait:301 - No account available for queue "SearchTimeline". Next available at 07:43:25 I still have this error |
Yeah, also happens to me, it's strange because it hits the 404 unexpectedly, sometimes it does sometimes it doesn't. It's not the most graceful handling, but I also hardcoded the account lock time in that scenario to just 5 seconds instead of 15 min |
Apparently the error 520 has to do with cloudflare not being able to correctly process the server's response, so I don't know if it has to do with this specific library. I never got that 520 error before |
Yep, just deactivated the lock too I'm investigating, will let you know if I find something |
I think the problem comes from the fact that with each request we refresh the page, instead of loading the home page once per account (this is the normal behavior of an X user) If you go to X in your browser, you'll sometimes see that you also have a 404 bug, you have to refresh for your session to work We'd need the same thing to solve the problem, one session per account + load the home page until it works, then we can generate our transaction ids without any problem Why do other X scrapers work? Because they don't have a pool system like we do, so they load the home page once, and then there's no problem (if it's successful). @vladkens Master if you have some time, we'd need your help to fix it |
@Rachetage2 Which other X scrapers? |
Like twikit, but twscrape >> with its pool system |
twkit has transaction id implemented: https://github.com/d60/twikit/blob/main/twikit/x_client_transaction/transaction.py#L141 Could be brought here who ever is working on it |
Yeah they basically implemented what that guy on the repo i posted did but async, could be easily ported in this repo actually... |
Any info about the 404 that often appears? |
not working on it atm as i already "stabilized" my service with the temporary patch i suggested yesterday, will look more into it when i got time |
Ok man, thanks One solution would be to only load the homepage one time, if the info already made one valid tid, it only ba valid ones after, so generate, if 404, generate again, if it works, save home page response to reuse it |
Hi everyone, I see this error with Thanks to @BonifacioCalindoro for investigating the problem and @fa0311 for reverse engenering the key generation algorithm. References: |
Love you man |
@vladkens if you're interested, I've been working on making the following library's implementation of the reverse-engineered process described in the https://antibot.blog/posts/ website work with twscrape. The implementation is here: EDIT - just read the thread above and it makes reference to this repo. Sorry for redundant comment ! |
I added this very strange code to generate pip install --upgrade twscrape |
Hi @vladkens , thanks |
Can we have twikit implementation ported to twscrape? They don't have the 404 issue somehow... |
I'm not sure how you did it, but If we scrape home page every request, that's why it's not stable. |
On my side it's pretty stable for the moment (not using limit btw), thanks for the fix, it's highly appreciated, won't forgive to support you after launching my project |
It seems the 404 problem occured because of problems in tid calcul, they resolved it in the new version. |
Anyone having an issue with SearchTimeline? Unhandled API response code: 404
The text was updated successfully, but these errors were encountered: