At 9:20am, 12:10pm and 5:20pm we experienced severe network degradation on our Cogent leg. No customers were impacted on our AT&T leg.

The first episode lasted 10min, the second lasted an hour and the 3rd lasted 20 minutes. At no time the network was down, but the degradation was so severe that some customers lost enough packets that they were practically offline.

Inside Valley Internet we could not detect any issues. Upon calling Cogent they acknowledged seeing packet loss but they could not identify the source.

As with our AT&T circuit that uses AT&T as a local carrier, so does our Cogent circuit uses AT&T as their carrier. Cogent then opened a ticket with AT&T and magically the network was in full swing within a few minutes. However, as usual, AT&T did not provide an explanation to the resolution.

To add insult to injury, the degraded connection caused many DNS requests to google to drop, causing google to treat our network as a source of an attack on their DNS…

Thus, while AT&T fixed their issue with Cogent, many of our customers who depend on google DNS to resolve internet addresses, were not able to get a resolution even though their connection was already functioning ok.

At this point and unless it reoccurs, we will need to treat it as another one of those infrequent AT&T outages which we’ll never learn or know what triggered it.