User Tweets API is returning 404

[12joan]

Probably related to zedeus/nitter#983

The API request to fetch Tweets for a given user is returning a 404 status and no response body.

twitter-client-web-1    | Using credentials: developer.twitter.com
twitter-client-web-1    | Guest token: Cache hit
twitter-client-web-1    | User ID: Cache hit
twitter-client-web-1    | Response {
twitter-client-web-1    |   [Symbol(realm)]: null,
twitter-client-web-1    |   [Symbol(state)]: {
twitter-client-web-1    |     aborted: false,
twitter-client-web-1    |     rangeRequested: false,
twitter-client-web-1    |     timingAllowPassed: true,
twitter-client-web-1    |     requestIncludesCredentials: true,
twitter-client-web-1    |     type: 'default',
twitter-client-web-1    |     status: 404,
twitter-client-web-1    |     timingInfo: {
twitter-client-web-1    |       startTime: 6269.856628000736,
twitter-client-web-1    |       redirectStartTime: 0,
twitter-client-web-1    |       redirectEndTime: 0,
twitter-client-web-1    |       postRedirectStartTime: 6269.856628000736,
twitter-client-web-1    |       finalServiceWorkerStartTime: 0,
twitter-client-web-1    |       finalNetworkResponseStartTime: 0,
twitter-client-web-1    |       finalNetworkRequestStartTime: 0,
twitter-client-web-1    |       endTime: 0,
twitter-client-web-1    |       encodedBodySize: 0,
twitter-client-web-1    |       decodedBodySize: 0,
twitter-client-web-1    |       finalConnectionTimingInfo: null
twitter-client-web-1    |     },
twitter-client-web-1    |     cacheState: '',
twitter-client-web-1    |     statusText: 'Not Found',
twitter-client-web-1    |     headersList: HeadersList {
twitter-client-web-1    |       cookies: [Array],
twitter-client-web-1    |       [Symbol(headers map)]: [Map],
twitter-client-web-1    |       [Symbol(headers map sorted)]: null
twitter-client-web-1    |     },
twitter-client-web-1    |     urlList: [ URL {} ],
twitter-client-web-1    |     body: { stream: undefined }
twitter-client-web-1    |   },
twitter-client-web-1    |   [Symbol(headers)]: HeadersList {
twitter-client-web-1    |     cookies: [
twitter-client-web-1    |       'guest_id=v1%3A169208846826427247; Max-Age=34214400; Expires=Sat, 14 Sep 2024 08:34:28 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None'
twitter-client-web-1    |     ],
twitter-client-web-1    |     [Symbol(headers map)]: Map(13) {
twitter-client-web-1    |       'date' => [Object],
twitter-client-web-1    |       'perf' => [Object],
twitter-client-web-1    |       'server' => [Object],
twitter-client-web-1    |       'set-cookie' => [Object],
twitter-client-web-1    |       'cache-control' => [Object],
twitter-client-web-1    |       'content-length' => [Object],
twitter-client-web-1    |       'x-transaction-id' => [Object],
twitter-client-web-1    |       'x-rate-limit-limit' => [Object],
twitter-client-web-1    |       'x-rate-limit-reset' => [Object],
twitter-client-web-1    |       'x-rate-limit-remaining' => [Object],
twitter-client-web-1    |       'strict-transport-security' => [Object],
twitter-client-web-1    |       'x-response-time' => [Object],
twitter-client-web-1    |       'x-connection-hash' => [Object]
twitter-client-web-1    |     },
twitter-client-web-1    |     [Symbol(headers map sorted)]: null
twitter-client-web-1    |   }
twitter-client-web-1    | }

[us3r1d]

Nitter has a possible solution: zedeus/nitter#983 (comment)

I don't have any recent experience with Twitter's API, so I don't grok the details; do you think it's something this twitter-client could pursue?

[12joan]

I've been keeping an eye on that issue thread.

I don't have any more experience with Twitter's internal APIs than you do (and possibly a lot less than some people who might be reading this), so I wouldn't know how to implement that solution myself. Once the specific API endpoints are shared publically, I can adapt it to JavaScript.

We might be able to make twitter-client interoperable with the tool for fetching guest credentials via proxies once that's released, configurable with a URL in the env variables or something. As a fallback for low-traffic clients, we can also include code to fetch credentials directly from Twitter (unproxied).

If anyone reading this knows how to implement the solution without waiting for Nitter, let us know and we can collaborate on an implementation.

[12joan]

This Nitter commit contains information on how to use guest accounts instead of guest tokens to access Twitter APIs. What we're still missing is concrete information on how to generate guest accounts programatically.

Here's the Nitter PR to subscribe to for updates: zedeus/nitter#985

If anyone wants to start working on this, remember that Nitter is AGPL-3.0 and we're public domain, so we can't use any copyrightable information from Nitter. (Nitter doesn't hold the copyright to Twitter's API interface, so this is fair game. Likewise, the general algorithm for interacting with Twitter's API doesn't belong to Nitter, but Nitter's implementation details do.)

[us3r1d]

Nitter sites can get guest accounts from a service that was set up for Nitter hosts at https://twitterminator.x86-64-unknown-linux-gnu.zip; I just re-activated my Nitter instance using that and it's working.

I really want to switch back to using this app to backend my Mastodon gateway, though; it's much more lightweight and the formatting of the posts is much better. :-)

[12joan]

That looks like a really nice service for fetching guest accounts, especially for people like yourself who already have it configured for a Nitter instance.

I want 12joan/twitter-client to be as flexible as possible with regard to sourcing guest accounts. I think the most extensible solution would be to expose an environment variable GET_GUEST_ACCOUNTS_CMD that can be populated, for example, as cat guest_accounts.jsonl or curl https://twitterminator.x86-64-unknown-linux-gnu.zip/download?[...]. Any shell command that outputs jsonl data to stdout.

Currently, I don't have any easy way of getting access to guest accounts to test with, nor do I have much time to spend on this at the moment. @us3r1d Are you able to adapt 12joan/twitter-client for compatibility with guest accounts, and also to add the environment variable described above?

[us3r1d]

Hah; I wish. I'd have done it already. :-)

The GET_GUEST_ACCOUNTS_CMD is a nifty idea for how to approach it, though.

I'll look at the source, but I probably don't have the knowledge to do this in the time I can give to it right now.