PHP Avoiding the Embed Timeline Twitter(status 429) Too Many Requests via cURL

[durangod]

Hi,

I have been working at this for about a week now. I have the twitter limit header data loaded via cURL and i am getting the embeds. The problem comes when i am testing the process and i have something i do not understand.

Here is the situation.

I am using this format with cURL which works fine when not 429 blocked. I am trying to test my error control process when it is blocked and that is when i discoved this situation.

I send this via cURL

https://syndication.twitter.com/srv/timeline-profile/screen-name/foxandfriends

The problem is that when blocked 429 i go to the console network tab which shows the url has been converted to a very long url by the twitter js i am guessing, and that long url is the one that has been blocked 429 shown both in the console and network tab (when it is blocked).

So to test i copy paste that long url and send it in cURL and i get a status code 200, and i am like WTH.... 🤬🤬

I dont get why the CONVERTED url is status 200 when i send it via cURL but it is 429 when the above url is send via cURL. i dont get it...
I thought maybe it had something to do with it being a GET request, so i set the cURL option to force GET even though that is the cURL default.

But the same issue.....

If you run that url above from a test cURL file, and check the network or console tab after the first time, you will see that it is blocked 429 and has been converted to a very long url. Now copy that same long url from the console or network tab and send that via cURL and you will get status 200 (and not 429).
WHY i ask !

Thanks 🙂

 

[Antero360]

Hi,

I have been working at this for about a week now. I have the twitter limit header data loaded via cURL and i am getting the embeds. The problem comes when i am testing the process and i have something i do not understand.

Here is the situation.

I am using this format with cURL which works fine when not 429 blocked. I am trying to test my error control process when it is blocked and that is when i discoved this situation.

I send this via cURL

https://syndication.twitter.com/srv/timeline-profile/screen-name/foxandfriends

The problem is that when blocked 429 i go to the console network tab which shows the url has been converted to a very long url by the twitter js i am guessing, and that long url is the one that has been blocked 429 shown both in the console and network tab (when it is blocked).

So to test i copy paste that long url and send it in cURL and i get a status code 200, and i am like WTH.... 🤬🤬

I dont get why the CONVERTED url is status 200 when i send it via cURL but it is 429 when the above url is send via cURL. i dont get it...
I thought maybe it had something to do with it being a GET request, so i set the cURL option to force GET even though that is the cURL default.

But the same issue.....

If you run that url above from a test cURL file, and check the network or console tab after the first time, you will see that it is blocked 429 and has been converted to a very long url. Now copy that same long url from the console or network tab and send that via cURL and you will get status 200 (and not 429).
WHY i ask !

Thanks 🙂

Hi there,
So as the code status says, HTTP response status 429 means that you are sending too many requests at a time... so, here is my question to you: What exactly are you attempting to do? You shouldn't be getting a 429 unless you are sending a flood of requests at a time.

 

[durangod]

Hi there,
So as the code status says, HTTP response status 429 means that you are sending too many requests at a time... so, here is my question to you: What exactly are you attempting to do? You shouldn't be getting a 429 unless you are sending a flood of requests at a time.

Hi,

The end goal is to display my twitter embeds.

My twitter embeds quit working statsus 429, so in my digging for answers i discovered that twitter added rate limits to their embed widgets. This is a site in development and i had taken some time away so i did not notice they had changed things until now.

So rather than get empty twitter embed i decided to try to trap the 429 and issue a message to the user that there is a timeout and the embed can be reloaded at a certain time. Like so for example:



The problem with the widget embed and rate limits is that because of the way their js works, it bounces back and forth
quite a few times to load the embed and all of those are counted as requests so just one widget load can exhaust the allowed
requests. I believe the default is 30 requests in 15 min, well that is just one widget load.

Here is an example: This is just one embed, and i have 3. You can see how one widget request can use up close to all 30 allowed.




Now im sure your question would be, how do you get the twitter data on limits.

That information is provided by requesting custom header data in the embed request via cURL

//custom headers for twitter rate limit values
        //x-rate-limit-limit: rate limit ceiling for the endpoint
        //x-rate-limit-remaining: remaining requests for the 15-minute window
        //x-rate-limit-reset: remaining time before the rate limit resets (in UTC epoch seconds)
        
        
        //request custom header values from twitter
        
        $custom_header = [
                          'x-rate-limit-limit:',
                          'x-rate-limit-remaining:',
                          'x-rate-limit-reset:'
                         ];

Those values are returned in the header from twitter in the cURL request.



So i have the rate limit data, now i just need to trap the 429 so i can show my custom message and not a blank widget container.

That is what i am trying to do is to trap the 429. The problem is that the same url that gets the 429 shows as status 200 when i specifically request it. So i dont see how i am going to trap the 429 if i cant get the same results.

During the request, twitter converts the request via their js to a very long url. That is the url that gets the status 429. So i test the status by having a test file request that url from the same domain but i get get status 200.

That is what baffles me, how can it be both 429 and 200 at the same time. Unless the js is manipulating the response out of site and issuing the 429 page but the actual url is 200. That is the only way it makes sense.


Does that make better sence now. I am not sure how i am going to trap the 429 now. 🙂

 

[durangod]

Here is an example:

Here is one request for one widget



That url is: (i have removed my domain and private info from it) domain and sub dir

https://syndication.twitter.com/srv/timeline-profile/screen-name/foxandfriends?dnt=true&embedId=twitter-widget-0&features=eyJ0ZndfdGltZWxpbmVfbGlzdCI6eyJidWNrZXQiOltdLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X2ZvbGxvd2VyX2NvdW50X3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9iYWNrZW5kIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19yZWZzcmNfc2Vzc2lvbiI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfZm9zbnJfc29mdF9pbnRlcnZlbnRpb25zX2VuYWJsZWQiOnsiYnVja2V0Ijoib24iLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X21peGVkX21lZGlhXzE1ODk3Ijp7ImJ1Y2tldCI6InRyZWF0bWVudCIsInZlcnNpb24iOm51bGx9LCJ0ZndfZXhwZXJpbWVudHNfY29va2llX2V4cGlyYXRpb24iOnsiYnVja2V0IjoxMjA5NjAwLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X3Nob3dfYmlyZHdhdGNoX3Bpdm90c19lbmFibGVkIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19kdXBsaWNhdGVfc2NyaWJlc190b19zZXR0aW5ncyI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdXNlX3Byb2ZpbGVfaW1hZ2Vfc2hhcGVfZW5hYmxlZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdmlkZW9faGxzX2R5bmFtaWNfbWFuaWZlc3RzXzE1MDgyIjp7ImJ1Y2tldCI6InRydWVfYml0cmF0ZSIsInZlcnNpb24iOm51bGx9LCJ0ZndfbGVnYWN5X3RpbWVsaW5lX3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9mcm9udGVuZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9fQ%3D%3D&frame=false&hideBorder=false&hideFooter=false&hideHeader=false&hideScrollBar=false&lang=en&limit=7&origin=https%3A%2F%2F

example.com%2Ftestdir

%2Findex.php&sessionId=f723831f138f277eff2d0254beb34c3680a37336&showHeader=true&showReplies=false&theme=dark&transparent=false&widgetsVersion=2615f7e52b7e0%3A1702314776716

Now when i run that same url i get a status 200, so how can it be 429 and 200 at the same time unless twitters js is doing something to manipulate the response?

 

[durangod]

So after a few weeks going over this i have come to the conclusion that for X (twitter) embed timelines you really only have a few options.

1. Use a third party service (mostly paid) to embed the timelines on your website, they do all the heavy lifting and you also avoid the rate limits.

2. Use the full X (twitter) API rather than their publish tool, the publish tool is very limited (rate limits) so youll have to build the whole API process using their API docs. I have heard they just released some improvements on this API so maybe its not that challenging. Who knows, honestly i dont think X(twitter) really cares anymore about making this ability user friendly for site owners.

3. Use an alternative to X(twitter), they are not the only project on the block, there are others in business.

4. Scrap the feeds and move on to other features, i think sometimes we all make things more critical than it really is 🙂