What Can I Do To UNDERSTAND Why I Can’t Reach CentOS.org (but Everyone Else Can)?

Home » CentOS » What Can I Do To UNDERSTAND Why I Can’t Reach CentOS.org (but Everyone Else Can)?
CentOS 28 Comments

What can I do to UNDERSTAND why, periodically, I can’t reach www.CentOS.org (but everyone else can)?

Every few months, http://www.CentOS.org just drops off for me and for me only. It’s not the web site, because I can reach it via a proxy (e.g., TOR browser bundle) and other tests show it to be alive (e.g., http://www.downforeveryoneorjustme.com/CentOS.org or http://traceroute.monitis.com).

But, for me, I just can’t reach it by any means.

Mike Easter kindly determined “CentOS.org does not echo ICMP or UDP
pings, only TCP, and its associated netblock level3 doesn’t echo UDP, but does echo ICMP and TCP.”

But, for me, this is what a traceroute returned just now on CentOS:

$ traceroute www.CentOS.org
… early hops removed …
16 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 211.850 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 205.221 ms ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 207.392 ms
17 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 128.453 ms ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 160.451 ms ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 160.437 ms
18 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 165.923 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 165.920 ms 178.414 ms
19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 241.537 ms 244.324 ms ae-3-80.edge3.Dallas1.Level3.net (4.69.145.136) 244.318 ms
20 * LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 244.291 ms 244.250 ms
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Here is a screenshot (modified to remove early hops) using mtr:
$ mtr www.CentOS.org
http://www2.picturepush.com/photo/a/12303130/img/12303130.png

My related questions are:
1. What can I do (on CentOS) to UNDERSTAND this problem?
2. Why does this problem happen every few months to me (and to just me)?
3. How can the net be this arbitrary and still work?

28 thoughts on - What Can I Do To UNDERSTAND Why I Can’t Reach CentOS.org (but Everyone Else Can)?

  • Rock wrote:

    Could you define “me”? Are you using this at home, or at work? What kind of connection? If this is work, are you paying for a fix IP?

    Interesting. I get to 20 (except for me it’s 15), and http://www.CentOS.org is the next hop (16). Now, I *do* have fat pipes and good connections (this is a US fed gov’t site), but still….

    Have you spoken to your ISP? When you try the traceroute, and it fails, have you tried other well-known locations, like google.com, or slashdot.org?

    Also, have you compared the contents of /etc/resolv.conf now, and again when CentOS.org isn’t visible?

    Sounds like either a firewall, spam, or nameserver issue.

    mark

  • thats a rather high hop count to get that far. I see…

    $ traceroute http://www.CentOS.org traceroute to http://www.CentOS.org (72.232.194.162), 30 hops max, 40 byte packets
    1 ge-0-0-1-414.er2.sjc1.got.net (207.111.214.242) 0.511 ms 0.440
    ms 0.462 ms
    2 ge-0-0-1-10.cr1.scz1.got.net (207.111.198.49) 3.961 ms 3.956 ms
    4.193 ms
    3 ge-1-3-0-745.cr1.sfo1.got.net (207.111.219.131) 7.796 ms 7.784
    ms 7.765 ms
    4 vlan114.car2.SanFrancisco1.Level3.net (4.53.130.89) 28.162 ms
    28.165 ms 29.320 ms
    5 ae-2-4.bar2.SanFrancisco1.Level3.net (4.69.133.158) 8.636 ms 8.621
    ms 8.594 ms
    6 ae-6-6.ebr2.SanJose1.Level3.net (4.69.140.154) 50.823 ms 50.664
    ms 50.917 ms
    7 ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 50.709 ms 50.545
    ms 50.533 ms
    8 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 50.560 ms 50.566
    ms 50.543 ms
    9 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 50.524 ms 50.418 ms
    50.407 ms
    10 ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 50.373 ms 50.310 ms ae-63-63.csw1.Dallas1.Level3.net (4.69.151.133) 56.827 ms
    11 ae-4-90.edge3.Dallas1.Level3.net (4.69.145.200) 50.268 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 50.282 ms ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 50.335 ms
    12 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 50.386 ms 52.124
    ms 51.458 ms
    13 http://www.CentOS.org (72.232.194.162) 51.202 ms !X 51.118 ms !X 51.113 ms !X

    which is only 8 hops to that same LosAngeles1 gateway.

    anyways, I never have any problems seeing it, so my guess is, your problem is in the first 15 hops you didn’t show.

    btw, the traceroute worked from my host using both -I and -T (icmp and tcp respectively) traces.

  • I have a fixed IP address at home.

    After hashing this out on alt.comp.freeware all morning, someone surmised that my static public IP address might have been added to a blocklist in the past few days, either by the penultimate hop or by CentOS.org itself.

    So, the question morphs to whether or not we have tools at our disposal which can pinpoint which entity is blocking my IP address?

    Note: I left a phone message at the DNS registrant for the penultimate hop and I left an email for the CentOS.org webmaster.

    Well, I must admit I never even knew that file existed. Here is what it has in it:

    $ cat /etc/resolv.conf
    # Generated by NetworkManager
    nameserver 192.168.1.1

  • Assuming you’re in Santa Cruz, I’m not more than 20 miles from you, but, I’m using a mountain WISP (with radio relays on all the hilltops), so that’s likely why I have many more hops than you do.

    Here is my traceroute with a bit more detail (my local stuff is redacted, for privacy).

    $ date Thu Feb 28 12:54:47 PST 2013

    $ traceroute http://www.CentOS.org traceroute to http://www.CentOS.org (72.232.194.162), 30 hops max, 60 byte packets
    1 router (192.168.1.1) 27.845 ms 27.807 ms 27.787 ms
    2 REDACTED STATIC IP ADDRESS 27.766 ms 27.745 ms 27.725 ms
    3 10.10.0.1 (10.10.0.1) 45.639 ms 51.532 ms 51.523 ms
    4 10.52.0.1 (10.52.0.1) 51.505 ms 53.765 ms 53.759 ms
    5 10.30.0.1 (10.30.0.1) 59.289 ms 62.681 ms 62.672 ms
    6 10.0.0.1 (10.0.0.1) 65.632 ms 114.790 ms 124.655 ms
    7 REDACTED WISP 127.280 ms 127.282 ms 127.266 ms
    8 REDACTED WISP backbone 127.247 ms 121.642 ms 48.416 ms
    9 REDACTED WISP backbone 51.977 ms 55.483 ms 55.473 ms
    10 REDACTED WISP backbone 64.521 ms 67.369 ms 67.352 ms
    11 xe-9-0-0.edge1.SanJose3.Level3.net (4.68.110.49) 64.463 ms 67.261 ms xe-11-1-0.edge1.SanJose3.Level3.net (4.68.111.189) 69.552 ms
    12 vlan60.csw1.SanJose1.Level3.net (4.69.152.62) 88.431 ms vlan70.csw2.SanJose1.Level3.net (4.69.152.126) 91.063 ms vlan90.csw4.SanJose1.Level3.net (4.69.152.254) 90.976 ms
    13 ae-62-62.ebr2.SanJose1.Level3.net (4.69.153.17) 96.160 ms ae-61-61.ebr1.SanJose1.Level3.net (4.69.153.1) 90.955 ms ae-82-82.ebr2.SanJose1.Level3.net (4.69.153.25) 90.896 ms
    14 ae-5-5.ebr1.SanJose5.Level3.net (4.69.148.137) 76.111 ms ae-2-2.ebr2.SanJose5.Level3.net (4.69.148.141) 89.501 ms ae-5-5.ebr1.SanJose5.Level3.net (4.69.148.137) 76.081 ms
    15 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 76.597 ms 76.026 ms 82.326 ms
    16 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 82.317 ms ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 95.557 ms ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 95.517 ms
    17 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 95.466 ms 97.991 ms 95.449 ms
    18 ae-83-83.csw3.Dallas1.Level3.net (4.69.151.157) 97.945 ms ae-1-60.edge3.Dallas1.Level3.net (4.69.145.8) 97.927 ms ae-93-93.csw4.Dallas1.Level3.net (4.69.151.169) 95.342 ms
    19 ae-4-90.edge3.Dallas1.Level3.net (4.69.145.200) 97.833 ms LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 97.845 ms ae-3-80.edge3.Dallas1.Level3.net (4.69.145.136) 62.875 ms
    20 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 77.207 ms 77.174 ms *
    21 * * *
    22 * * *
    23 * * *
    24 * * *
    25 * * *
    26 * * *
    27 * * *
    28 * * *
    29 * * *
    30 * * *

    What I now know (that I had not known prior), is that the hop you see at 20 is the penultimate hop.

    So either hop 20 is not forwarding the packets, or the final destination is not acknowledging them.

    Someone suggested I might be on a blacklist, somehow, and that, at least, would make technical sense.

  • Rock wrote:
    Oh. great. Have any bouncing mail when CentOS.org disappears? I have this happen a number of times a year, and yell about it every time. They use dnsorbs, who I say use a blacklisting methodology that worked 15 years ago, but is a complete disaster now, when one hosting co (as I have now), or ISP (say, roadrunner in Chicago, which provides one third or one half of *all* of Chicago’s net access), and both have mail go through one mailserver address… and when a few idiots get their systems or sites compromised, dnsorbs instead of blocking each domain, blocs ALL the hundreds of thousands of folks innocently emailing through their service provider or hosting site. They don’t seem to parse headers, and go *past*
    the mailserver to the culprit. Thank you, I prefer fail2ban’s methodology.

    Right, that’s given you by your cable modem or router. Can you see what that is on that ISP-provided box? Alternatively, add well-known nameservers to that file, which will be overwritten when it’s working again. For example, you could add nameserver 8.8.8.8
    nameserver 8.8.4.4
    (see < https://developers.google.com/speed/public-dns/docs/using>)

    mark

  • you can’t parse the headers until you read them, and you can’t read the headers until you accept the incoming message. once you’ve accepted it, you can’t bounce it back to the sending server via refusing the connection, and if you try and bounce it to the ‘from’ address you’ll be spamming a lot of innocent parties who’s email addresses have been forged on said headers.

    so, if you use header parsing, all you can do is quietly drop the message.

  • The only blacklists I know about involve mail and require the receiving machines to actively choose to do lookups and reject based on the result. Nothing should be dropping/blocking other types of traffic. There are an assortment of ‘looking glass’ sites on the internet where you can originate traceroutes and check that BGP routes for source and targets are being propagated. This seems to be a directory of those sites: http://www.lookinglass.org/

  • That’s not true – there are several phases to SMTP delivery and you can reject at most of them. You just have to have a mailer where you can control the operations at the right place. Using sendmail with MimeDefang as a milter gives you about as much control as possible if you want to parse/scan headers and content and react accordingly.

  • Dunno the exact number of hops but it’s always long but I only have 1 WISP so I don’t have a choice.

    It goes from my laptop to my home broadband router to my radio & antenna on the roof to the WISP antenna and at that point it bounces a while within the WISP’s network before exiting on some backbone on it’s merry way to CentOS.

    The ONE STOP prior to CentOS.org is the last stop; but all other web sites work. So, my “assumption” is that CentOS.org is blocking me. I didn’t think about this when I first posted this thread, but then someone told me that they ran a traceroute and there was only ONE STOP after where it stopped for me, so, that’s when the light bulb lit.

    Now, as to WHY they’re blocking me – that I asked their webmaster by mail, earlier today.

    Nice site. I see blacklists, whitelists, Yellowlists, neutralized, and even not listed as reports, and ALL (even the not listed) came up with zero.

  • Very true. I thinking that if he _was_ on an email blacklist, then his network might _also_ have been flagged as bad (by something else). Because I didn’t know about …

    Hey, that is very cool. Did not know such a thing existed.

    Thanks for pointing that out ;-)

    K

  • Am 28.02.2013 23:52, schrieb John R Pierce:

    Not true. You can read the entire mail in the SMTP DATA phase and still reject it after the terminating single dot. Works perfectly fine on several MIMEDefang installations I set up to reject incoming mails containing malware or exceeding a certain SpamAssassin score.

    HTH
    T.

  • For the record, I was unable to get to any of these long-hop destinations:
    $ traceroute tempotv.com.tr ==> died on the 23rd hop
    $ traceroute -I http://www.CentOS.org ==> died on the 19th (penultimate) hop
    $ traceroute http://www.gu.ac.ir ==> Iran, died on the 22nd hop
    $ traceroute http://www.gu.edu.pk ==> Pakistan, made it only to the 19th hop
    $ traceroute -m 255 http://www.tourism.gov.my ==> Malaysia died on the 19th hop

    All the above died before reaching their destinations.

  • Tilman Schmidt wrote:

    Right, I meant to respond to that, but forgot before I got home.

    Look, somehow, someone, somewhere, has to decide they’re receiving spam from an address… and the question is, *what* address. By trying to block what are allegedly “open relays”, they’re *also* blocking very large hosting and service providers, *all* of whose mail goes through that gateway. What *should* be reported to be blocked is the domain that’s sending the spam.

    Blocking an open relay should be done *only* on human investigation, to see whether that’s the majority of what’s coming out of there, and consideration of what the “relay” is, whether it’s a known source, or an innocent large provider.

    mark

  • Rock wrote:
    Back to the question: have you spoken to level two, at least, tech support of your provider of network access, to see if they have some explanation?

    mark

  • Note that traceroute requires a round-trip for you to see success. A
    failure is almost as likely to mean that the router where it stops does not have a route back to the source as that it can’t reach the next forward hop. Those ‘looking glass’ sites can show you if the bgp routes are propagating to various locations.

  • Hi Mark,

    My WISP provider is as perplexed as I because I’m the first subscriber to have this problem, he says.

    He thinks it’s on the CentOS side of things.

  • The argument on the other side of this is that the blacklist maintainers don’t really block anything, they just make a lookup service available that corresponds to certain reported or potential problems. It is the mail delivery service provider that makes the decision to reject mail (or not) on behalf of the recipient and no one forces them to use the blacklists. If the recipient doesn’t want this, they can use some other mail service or (sometimes) set up whitelists. A couple of decades ago I would have agreed that no one should reject but I think we’ve learned that you can’t trust everyone with an IP address.

  • Some of the CentOS mirrors sites are insanely fast – maybe when they are syncing stuff out it overloads some intermediate connections.

  • I saw the reference to the lookingglass site, e.g., http://www.lookinglass.org But, I’m sorry … I’m totally clueless as to how to properly USE them.

    May I ask:
    Q: Which of the IP addresses below do I put into lookingglass to gather data?

    Given that these all died as follows:
    $ traceroute tempotv.com.tr ==> died on the 23rd hop
    19 82.222.13.145 (82.222.13.145) 262.456 ms 82.222.3.117 (82.222.3.117) 270.129 ms 273.278 ms
    20 82.222.13.106 (82.222.13.106) 276.814 ms 276.812 ms 276.797 ms
    21 82.222.253.82 (82.222.253.82) 273.131 ms 264.450 ms 247.432 ms
    22 82.222.254.210 (82.222.254.210) 247.354 ms 247.362 ms 82.222.224.234 (82.222.224.234) 309.858 ms
    23 asy60.asy53.tellcom.com.tr (92.45.53.60) 326.022 ms 329.493 ms 329.483 ms
    24 * * * (and so on)

    $ traceroute -I http://www.CentOS.org ==> died on the 19th (penultimate) hop
    15 ae-6-6.ebr2.LosAngeles1.Level3.net (4.69.148.201) 60.128 ms 62.055 ms 63.873 ms
    16 ae-3-3.ebr3.Dallas1.Level3.net (4.69.132.78) 64.999 ms 56.867 ms 59.059 ms
    17 ae-73-73.csw2.Dallas1.Level3.net (4.69.151.145) 60.437 ms 62.928 ms 57.975 ms
    18 ae-2-70.edge3.Dallas1.Level3.net (4.69.145.72) 59.828 ms 55.071 ms 57.122 ms
    19 LAYERED-TEC.edge3.Dallas1.Level3.net (4.71.170.6) 56.049 ms 65.838 ms 58.644 ms
    20 * * * (and so on)

    $ traceroute http://www.gu.ac.ir ==> Iran, died on the 22nd hop
    14 rostelecom-ic-300487-ffm-b11.c.telia.net (213.248.96.202) 197.545 ms 197.545 ms 284.625 ms
    15 95.167.92.186 (95.167.92.186) 324.490 ms 324.486 ms 324.468 ms
    16 95.167.14.198 (95.167.14.198) 378.393 ms 378.393 ms 378.378 ms
    17 * * *
    18 195.146.33.29 (195.146.33.29) 369.322 ms 369.306 ms 300.357 ms
    19 78.38.245.228 (78.38.245.228) 300.263 ms 300.264 ms 306.080 ms
    20 78.38.244.250 (78.38.244.250) 290.148 ms 300.201 ms 300.198 ms
    21 89.221.87.25 (89.221.87.25) 338.022 ms 338.042 ms 337.997 ms
    22 95.38.120.254 (95.38.120.254) 330.673 ms 330.681 ms 330.669 ms
    23 * * * (and so on)

    $ traceroute http://www.gu.edu.pk ==> Pakistan, died on the 19th hop
    10 charter-peer.sv1.layer42.net (69.36.225.18) 45.929 ms 48.200 ms 48.191 ms
    11 bbr02snjsca-bue-6.snjs.ca.charter.com (96.34.3.2) 48.173 ms 50.328 ms 52.805 ms
    12 bbr01dnvrco-bue-2.dnvr.co.charter.com (96.34.0.3) 81.440 ms 81.442 ms 81.423 ms
    13 bbr01olvemo-bue-1.olve.mo.charter.com (96.34.0.145) 112.975 ms 112.900 ms 112.801 ms
    14 bbr01blvlil-tge-0-15-0-4.blvl.il.charter.com (96.34.0.111) 129.811 ms 129.811 ms 93.669 ms
    15 bbr01atlnga-tge-0-0-0-9.atln.ga.charter.com (96.34.0.120) 122.469 ms 122.468 ms 146.517 ms
    16 crr01sghlga-bue-3.sghl.ga.charter.com (96.34.2.71) 123.937 ms 123.943 ms 127.972 ms
    17 acr02atlnga-tge-0-0-0-0.atln.ga.charter.com (96.34.72.179) 130.189 ms 132.137 ms 132.136 ms
    18 75-131-187-34.static.gwnt.ga.charter.com (75.131.187.34) 129.996 ms 129.998 ms 129.978 ms
    19 67.23.161.143 (67.23.161.143) 129.894 ms 129.873 ms 129.859 ms
    20 * * * (and so on)

    $ traceroute -m 255 http://www.tourism.gov.my ==> Malaysia died on the 19th hop
    11 gi9-0-0.cr2.nrt1.asianetcom.net (202.147.50.134) 244.797 ms 244.804 ms 244.797 ms
    12 ge-0-0-0-0.gw3.nrt4.asianetcom.net (202.147.0.226) 215.347 ms 215.348 ms 220.710 ms
    13 61.8.59.22 (61.8.59.22) 267.994 ms 268.004 ms 267.960 ms
    14 ge-2-3-0-0.sin-64cbe-1a.ntwk.msn.net (207.46.41.75) 325.436 ms 259.629 ms 259.605 ms
    15 10.22.212.213 (10.22.212.213) 217.562 ms 10.22.212.209 (10.22.212.209) 273.695 ms ge-5-2-0-0.sin-64cbe-1b.ntwk.msn.net (207.46.41.26) 265.336 ms
    16 10.175.56.241 (10.175.56.241) 265.273 ms * 10.175.56.49 (10.175.56.49) 272.494 ms
    17 10.175.61.5 (10.175.61.5) 282.798 ms 10.175.60.7 (10.175.60.7) 248.975 ms 10.241.90.205 (10.241.90.205) 258.597 ms
    18 10.78.236.2 (10.78.236.2) 258.541 ms 10.78.220.2 (10.78.220.2) 211.475 ms 10.241.88.59 (10.241.88.59) 220.578 ms
    19 * * 10.78.246.2 (10.78.246.2) 239.054 ms
    20 * * * (and so on)

  • Rock wrote:

    Does tech support have the same problem?

    I mean, here on our internal network, some idiot blocked the BBC; after I
    put in a ticket, they unblocked it… then contacted me, and I found the main page was unblocked… but *every* article link was still blocked, and the guy I was talking to told me he could see them… but someone he was working with couldn’t…. They did, finally fix it, and claimed it had happened to to a phishing attack….

    mark

  • If you thing the problem is on your side, try ping/traceroute to your own IP from some other locations.

    It is very common for organizations to block ICMP at their private border routers and just let a few port/address combinations through for the public services they provide, so not reaching a target does not mean there is a problem. Sometimes you can reach the target with tcptracroute (or traceroute -T -p port), but some intermediate hops may not show because their ICMP replies are blocked.

    Organizations that are multi-homed and do their own BGP routing will have fairly specific routes showing up in the bgp tables that you can query from the looking glass tools and if they fall completely off the grid the routes will disappear.

  • Am 01.03.2013 17:38, schrieb Rock:

    Yeah, providers would say that. Several times in a row to different subscribers, if necessary.

    Indeed. I know at least three big providers hereabout for whom I
    routinely warn my customers if they have to call support that the first answer will invariably be: “Everything’s ok at our side, the problem must be with your equipment/your communication partner/anywhere but us.”

    Good support engineers don’t make statements about *where* the problem is before they have determined *what* the problem is.

    SCNR

  • Rock wrote:

    Your traceroute is UDP. None of those answer UDP. All but the last one will answer ICMP.

    If you use your mtr on them in default ICMP mode, all but the last will ICMP traceroute. The last one needs tcptraceroute.

    traceroute is not the best tool for some purposes. All but the last of those will ping ICMP.

  • We have a dynamic iptables blocking script on the site that adds IPs that do lots of nmap scans, try to do xss, spam the forums, etc. If you e-mail me your IP at johnny@CentOS.org off list, I’ll take a look and see if you are being blocked by that.

    Thanks, Johnny Hughes

LEAVE A COMMENT