JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[RasmusFonseca]

Running the minimal example crashes with an error:

$ ipython
Python 3.8.3 (default, May 19 2020, 13:54:14) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.17.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from twitter_scraper import get_tweets                                                                                        

In [2]: for tweet in get_tweets('twitter', pages=1): 
   ...:     print(tweet['text']) 
   ...:                                                                                                                               
---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-2-bd0d769d18f8> in <module>
----> 1 for tweet in get_tweets('twitter', pages=1):
      2     print(tweet['text'])
      3 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in get_tweets(query, pages)
    169             pages += -1
    170 
--> 171     yield from gen_tweets(pages)
    172 
    173 

~/opt/miniconda3/lib/python3.8/site-packages/twitter_scraper-0.4.3-py3.8.egg/twitter_scraper/modules/tweets.py in gen_tweets(pages)
     34         while pages > 0:
     35             try:
---> 36                 json_response = request.json()
     37                 html = HTML(
     38                     html=json_response["items_html"], url="bunk", default_encoding="utf-8"

~/opt/miniconda3/lib/python3.8/site-packages/requests/models.py in json(self, **kwargs)
    896                     # used.
    897                     pass
--> 898         return complexjson.loads(self.text, **kwargs)
    899 
    900     @property

~/opt/miniconda3/lib/python3.8/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    355             parse_int is None and parse_float is None and
    356             parse_constant is None and object_pairs_hook is None and not kw):
--> 357         return _default_decoder.decode(s)
    358     if cls is None:
    359         cls = JSONDecoder

~/opt/miniconda3/lib/python3.8/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/opt/miniconda3/lib/python3.8/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[Avishay116]

Same here - python 3.7.4

[gonhonoria]

same issue. python 3.7.6

[yimingStar]

The response might be empty due to the request failed.

[gonhonoria]

ok, Why would the request failed? I have tried with a bunch of account names but still getting the same error. Has there been any change on the script?

[gonhonoria]

While trying parsing from Hashtags, it works. The issue is specific to parsing profile tweets

[yimingStar]

It is because the url request for the user profile is somehow unavailable. See issue #166, the user-agent that this module is using cannot get the proper response now. Maybe Twitter has banned the user-agent, or maybe Twitter has changed all their website into Client Side Rendering causing request-html failed to get the elements that it usually could.

[botbotbotbotboot]

@yimingStar Agree. Looks like twitter has made some major changes.

[botbotbotbotboot]

If anyone interested in a workaround, please check this out @yimingStar @bisguzar

Foo-Manroot/tweet-feed#1 (comment)

Dont know if this method is good enough.

[dgnsrekt]

Here is my attempt at a solution based on @Foo-Manroot instructions.
https://gist.github.com/dgnsrekt/8cdb0c10e4a8fe6dcef65ee3a66a4e33
I'm having an issue with getting rate limited. Once, I can get past that issue I can make it nice and pretty and submit a pull request.

[magicaltoast]

Same issue

[n0madic]

Found another source of tweets - widgets at https://publish.twitter.com/
Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME>
The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

[Nicknameinmel]

Found another source of tweets - widgets at https://publish.twitter.com/
Get JSON from https://syndication.twitter.com/timeline/profile?screen_name=<NAME>
The information there is incomplete, for example, there are no likes, retweets, etc. But there is no rate limit.

Could you tell me how to do that! Many thanks!

[n0madic]

Everything is very simple - we take JSON from the link, take from it the “body” field with the timeline and parse it as usual. JSON also has "minPosition" and "maxPosition" headers for pagination. Implemented in my Go project https://github.com/n0madic/twitter-scraper.

[victorDaniloCasta]

I have the same issue on Python 3.8.2.
Everyone know other library for scrapper or know If the new API twitter version allows to get tweets from more than seven days ago?
Thanks a lot.