More evidence that CenturyLink is broken

[MarkJFine]

I have a bot that runs three times a day at 7, 9:45, and 12 noon that pulls an RSS feed for the website I administer, embeds the last 10 article titles onto an image and posts that image to the site's Twitter account.

Two days ago one of those instances got a timeout while waiting for an oauth authentication with Twitter and failed. Immediately following that I manually logged in and was able to run the Twitter portion of the bot manually without error.

Yesterday, all three went fine.

Today, only two of those instances failed for the same timeout during oauth. I just logged in and ran fine manually.

Twitter's backbone is Cali is <drumroll> Level3/CenturyLink

Folks... all of this can't possibly be coincidence.

[MarkJFine]

Incidentally, just had a disturbing Twitter exchange with their social media 'help'... bottom line: they wouldn't do anything without a DM.

Doubtful they would have checked into anything anyway, since I don't have an account with them.

[gaines_wright]

  I've been doing a lot of expermenting  with traceroute today.  I'm just sort of stumbling around in the dark with this.  One of the thngs I've discovered is that the first 6 entries seem to be the same no matter where I'm tracing to:

1 192.168.42.1 (192.168.42.1) 0.223 ms 0.236 ms 0.274 ms                                                          //  Hughes router
2 100.67.205.153 (100.67.205.153) 0.905 ms 1.016 ms 1.169 ms                                                  //   Shared address 
3 dpc6935186182.direcpc.com (69.35.186.182) 589.663 ms 589.681 ms 589.643 ms                  //   Hughes site
4 dpc6935188058.direcpc.com (69.35.188.58) 1369.224 ms 1369.243 ms 1369.193 ms              //   Hughes site
5 ae13-575.edge3.Denver1.Level3.net (4.28.103.141) 1369.171 ms 1369.150 ms 1369.125 ms  //  CenturyLink

6 * * *

  The text after the C++ style comment is what I've  been able to discover about the address so far.   When CenturyLink appeared I remembered your post. 

[MarkJFine]

Yes. So this is what you're looking at:
1. Your wifi/router, sometimes these show your public IP depending upon the tracert options. Not sure why the modem itself (192.168.0.1) doesn't show.

2. Gateway input from satellite

(not sure why it's showing a shared IP on yours, should be something in 69.35.184.0/21 like the other two)

3. Gateway processing

4. Gateway output to internet (may vary)

5. Upstream provider to gateway

6. [is always obfuscated by ICMP blocking for some reason on CL, but should be the provider's backbone]

...

all the other nodes to the target address.

[gaines_wright]

Thanks for the info.   I guess the differences may be caused by my use of the Linux version of traceroute.

[MarkJFine]

Should be the same as my Mac... FreeBSD version.

[gaines_wright]

---

@MarkJFine wrote:

Should be the same as my Mac... FreeBSD version.

---

  Well, again thanks.   I may abandon my attempts to understand all of this, especially since today I did another trace to aol.com, and found that many of the hops past the first 6 had disappeared.  Yesterday I had several hops to telia, which is in Sweden, and hinet which is in Taiwan.  All gone today, aol.com went from 22 hops to 15 hops overnight.

   Ahhh!   It just occurred to me that the routing table could be constantly changing, and this makes the hops change also.  Question is, do I really need to know all of this stuff while retired and 72 years old?  I'll probably stick with it for a while, I've always loved useless information.  :>)>

[MarkJFine]

Yes, it is dynamic and CenturyLink's backbone controls much of the routing table by virtue of where it sends you first. Sometimes it's due to known down links. Sometimes it's because they're just that screwed up. Clearly my view is the latter.

[MarkJFine]

Seems like the random 1 minute drop outs are extending up to 2 minutes or more in some cases. Not only from what I see from SDO, but also on links between two CL servers (the Cloudflare LA server and a DigitalOcean server), and between my webserver in Pittsburgh and Twitter.

Also, just a thought, but the dropouts may feasibly be responsible for the spate of dropped calls I see being reported due to the link timing out.

[maratsade]

One instance when more is not better.  I hope someone's doing something about CL's issue(s). 

[MarkJFine]

Even more evidence:

Recently moved the bot off my server (with the 2-hops over CenturyLink) to the server/site it's supposed to operate on in Germany... Since I did that, zero timeout errors to Twitter.

Only problem is that for some dumb reason 1&1/ionos don't have traceroute installed on that box. This isn't the stupidest thing they've ever done (they use X-Forward IPs as the incoming IP, so you can't filter non-Cloudflare IPs, causing me big headaches), but it keeps me from seeing what path is actually being taken.

[maratsade]

I think CL is going to break down even more in the coming weeks....