Work from the library!
On the morning of July 8th, I started work as normal. My home internet connection was down, but my cellphone was not. I tethered my phone (before the network overloaded) to communicate with the office and discovered LARG*net was not affected.
It was a curious situation with my home internet connection. I had synchronization with my cable modem, but I could not obtain an IP address. Indicating the problem was not at my house. I checked the usual websites to see how wide the problem was. It was bad, it was nationwide. Thousands of users in Vancouver (around 4am their time) had already noticed they were offline.
We checked our monitoring systems to confirm our members were still online. As staff started arriving at their offices, bandwidth usage started to increase. Lucky for us, summer break is off peak usage with university, college and public-school students away. We had internet capacity to spare. London Health Sciences Center lost their public wi-fi to the outage. To compensate, LARG*net provided several gigabits of bandwidth to serve internet to the hospital visitors.
The London Public Library used social media to promote “Work from Library Day” to draw attention to their services. People responded and on average 1200 people were connected at any given time, quickly overloading internet capacity for the library branches. They called for assistance, and within minutes we turned them up to maximum capacity. We were able to donate significantly more bandwidth, which was used to help the London community stay connected. Thank you LPL, for the big shout out on Twitter!
What other ISP can do this for your organization on a moment’s notice? Let alone during a national emergency? We are a not-for-profit organization, here to work with our community. We never charged our members for helping out during a crisis. This is the secret sauce of LARG*net which no large ISP can ever match.
Everyone’s home internet goes down occasionally. Few people have a firewall with redundant internet connections at home. Did you know you can go over to your nearby library with your laptop and get free wifi? This is your personal internet redundancy.
Redundancy!
Redundancy is the name of the game. Anything can happen, and here are some reasons for outages (RFOs) we have encountered:
Squirrel, chewing through conduit and fiber optic cables.
Construction, backhoes digging up or crushing fiber conduits.
Vandalism, people cutting cables, hoping to sell copper for profit.
Trucks, or cars striking utility poles and damaging fiber optic cables.
Melting snow, flooding conduits to enter a building and short-circuiting network gear.
Image source unknown
What about LARG*net’s redundancy?
How about an insider view of how we stay online during major outages? Caveat: I work on cyber security and I’m no longer a network engineer. I don’t know the specifics, but we stayed online due to significant redundancy and capacity. As an ISP, we operate using carrier grade equipment. Imagine a router so large, it sounds like an airplane taking off at the airport.
Our many transit connections provide full internet routing tables from multiple carriers. Some local in London, others in cities such as Toronto or Chicago. This enables fast transit to all the big content providers directly, without relying on a sole provider. Our members will connect to our multiple datacenters, with diverse fiber paths to maintain resiliency. Is there an extra cost to this? Yes, but how would a communications outage affect your business? Especially if you provide critical services such as a hospital and utility company. If your business is in e-commerce, I’m sure you have calculated how much loss you incur for every minute of an outage.
As we return to the “new normal”, video conferencing and 4K video streams have become the new norm, requiring substantial bandwidth. Our members’ requirement for capacity grows, so must we. We are planning to branch out to yet another Internet Exchange point (IX), further extending our redundancy as we expand capacity. Montreal? Detroit? New York? We don’t know yet, as we are in the planning stages.
If a large Tier-1 ISP suffers an outage, it is not likely to affect our operations completely, or our members’ ability to connect to the internet. In fact, this happens quite often and that’s why we have redundancy to mitigate its effects. Hence why we stayed online when most of the country was down.
This is not to say we don’t have any take away from this incident. All our support staff now have dual SIMs on their cellphone, connecting them to two different carriers. What good is redundancy if you can’t communicate with each other?
Tristan Bienstman, Cyber Security Analyst - LARG*net