The military internet, which is built on similar principles and protocols, yes. The commercial internet? Ehhh. Not so much.
Most internet interconnects are consolidated into internet exchanges, usually in, or near, datacenters. I live in southern Ontario, pretty much every local ISP routes the majority of their traffic, if not all of it, through TorIX in Toronto. There’s a datacenter there, or more accurately several, which are cross connected into or through TorIX. Many of the local ISPs do not have redundancy with another IX or datacenter. One notable exception is Bell Canada, who has a strong, and broad transportation network of their own, which connects to TorIX and the IX in Montreal, among others. They’re probably the best set up in their distribution layers from what I can tell.
Most ISPs can suffer the loss of one major upstream network provider loss, but not all of them in the IX. Aka, if the datacenter/IX falls, the entire network goes offline.
Take for example a local third party provider of teksavvy. And to be clear, I’m making a lot of wild speculation about teksavvy here, and reality may be very different, but I’m just using them as an example. Normally, small ISPs like teksavvy do one of two things, they either resell DIA connections (direct internet access), which is basically just buying another providers access from them. Aka, you buy teksavvy, but the line you’re connected to is from Bell, your internet and your internet IP is provided by Bell, and you pay teksavvy who pays Bell. You might as well be on Bell, but since teksavvy is buying in bulk, you can usually get service from them cheaper than from Bell. Functionally it’s the same, so there’s a null sum here. The alternative is that teksavvy gets a wholesale connection to you, where Bell (or another “last mile” ISP) provides the connection from your premise to teksavvy, usually through a local IX, your connection still goes into the datacenter, but rather than go from the bell line, through Bell’s gear direct to the internet, it goes from Bell’s wholesale line, to teksavvy gear in the datacenter, then gets rerouted through to the internet from there. All within the DC/IX.
The IX crossconnects all providers in the DC, which would be companies like Meta, Google, Netflix, the company formerly known as Twitter, etc, and also to other ISPs, so you can directly connect to your friend down the road on Rogers without having your traffic go further than it has to. The IX is where ISPs, providers, datacenters and all other connections meet. A nearby nuclear blast that destroys the DC/IX in your area, would very likely disrupt all communication for the area served by ISP connections routed through that IX.
In rare cases (I believe Bell is set up this way), the ISP will cross link it’s distribution centers, which are usually buildings in your neighborhood with the company’s logo, but no customer facing area, to eachother all the way across an area into the next area where the next IX is. So you’re functionally connected to two IX locations or more. The providers distribution network isn’t as fast as the IX to IX direct fiber routes, which usually have fewer devices to go through, but it would work at very limited capacity.
On the public internet, there is a mesh between IX’s, but the location of the IX isn’t hidden. Any individual connection to the internet is usually only connected to a single IX. Most of the distribution between you and the IX is not redundant.
Then compare with what I believe would be a typical example of a military “internet” (again, I’m just speculating based on highly redundant data principles, nothing more): each location, like a military base, airfield, government complex, etc. Would be connected to multiple other locations in a mesh. In addition, they would have backup links likely through satellite or microwave relay links. So if all of the locations that a base is connected to go down, they can use satellite as a final option. Imagine nine such sites in a standard 3x3 grid. Each is connected to no less than 3 neighbors, and the central base is connected to all eight of the others. The central base is destroyed (or otherwise has their datacom disabled). All outlying bases still have at least two links to neighbors. Communication continues. Another base is destroyed, say, the north-East location. All nodes can connect to eachother without issue, but the north and east bases are down to one link plus backup. The South-Eastern location goes down. The rest are unaffected except the eastern base which now needs to rely on satellite backup, but they can communicate. Etc.
As you can hopefully see, the bases all have multiple redundant links and unless they are destroyed or otherwise have their datacom disabled, they can remain in communication with the other bases.
With the internet, that’s usually how IX’s are interconnected, but anything on a single IX, is basically fucked if that IX no longer operates. We’re all nodes of an IX. An intelligent adversary would target any known centers of telecommunications, and we’ve made it easy for them my centralizing all of the communications for a given geography into centralized locations which are conveniently published for anyone to see. There are lists of IX locations on the internet for anyone to stare at, including what ISP companies are connected. With a single well-placed, high yield (not even nuclear) bomb or ICBM, all commercial communication systems can be neutralized for an area of attack. They wouldn’t even need to destroy a full city block to accomplish it.
Sure, bombing, say TorIX, wouldn’t stop someone in, say, Nova Scotia, from chatting to someone in British Columbia, but pretty much everyone in Southern Ontario would be disconnected in an instant. Unable to tweet about the bomb that just went off in downtown Toronto (I believe the IX there is next to Union station, at 151 Front St).
The reason it’s done this way is because of money. It’s far cheaper to centralize access to an IX, then connect the IX’s together. The alternative is to string fiber between datacenter buildings for different companies at different addresses. Getting fiber between geographically different locations is costly, so the companies at the DC/IX all pay a small portion of the fees, either directly or indirectly, to have each IX connected to the others. That cost is shared and the various providers can have access to a limited number of fiber strands that run between locations. Even a handful (like 5) strands can net about 1Tbps of bandwidth or more depending on how it’s used. The planning and deployment of such connections can easily run into the millions of dollars. Sometimes significantly more. Sharing in that cost is a good business move, even if you’re “helping” your competition by doing so, because they also are getting a benefit from it.
Taking it back to the OP: the blast observed appears to be in a city, where it is likely an IX would be situated, most likely the one that your shelter is connected to. Cellular won’t help since a lot of that infrastructure relies on microwave relays back to a head node, usually on the cities outskirts, which then transfers the data to a high-speed fiber line to… You guessed it, the IX.
The internet isn’t designed to be resilient to war. We will lose all datacom if war breaks out, with certainty.
Military networks, though similar in design, have vastly more redundancy, the likes of which, we, the people, do not, and will likely never, have access to.
IX vendor lists are pretty much public information. There’s plenty of ways to find the information but you have to work backwards from how you’re thinking. The way you’ve phrased this is to find which IXes an ISP is connected to, but you can only really find what ISPs are at which IXes.
For example, TorIX, a limited list is literally on their main page of their official site:
https://www.torix.ca/
Rinse and repeat for every IX.
If you can find which business segment for an ISP handles consumer connections, you might be able to find them on the peering db ( https://www.peeringdb.com ), which, if you dig hard enough, you could get a list of which IXes the ISP is linked to (aka, peered with). Taking a major ISP here as an example, Rogers, there are eight networks with “Rogers” in the name. A couple are probably not the Rogers ISP/Telco company, but it looks like six might be, each is some business segment. Now, I know that Rogers does most of their internet services using AS 812, so I’d start there, and I can see they’re linked to Toronto, Montreal, Vancouver, new York, Chicago, Seattle, and Ashburn. The question I can’t answer is, does Rogers have a non IX-IX link (something geographically divergent) from, say, Toronto, and their clients there, over to Montreal/new York/Chicago/(etc). Some kind of interconnect that doesn’t require the IX to exist? If I’m in, say, London, Ontario, about two hours west of Toronto, (where my closest IX would be Toronto), and the IX in Toronto goes away, will there be a way for Rogers to get my traffic out to the internet? Or does my traffic go to Toronto and get relayed from there every time?
There’s no possible way to know without having intimate knowledge of the interconnects that Rogers owns and where they are physically run. Even if that fiber link looks like it goes straight from the London area to, say, Chicago, how do you know that the physical fiber doesn’t go towards Toronto, land in the IX then get switched across an IX-IX link, to the Chicago IX? It’s impossible to know.
Turning it around on Montreal, if I’m located in say, Ottawa, which is pretty much midway between Montreal and Toronto (closer to Montreal), and the Montreal IX ceases to exist, will my traffic have a viable path to the Toronto IX? In the same way, I have no idea. It could bounce through Montreal’s IX before going to Toronto over an IX-IX link.
I suspect that the Ottawa situation is much more likely to have dedicated fiber running from there to both Toronto and to Montreal, since it’s the capital of Canada, and having that kind of geographically diverse path to a couple IXes, would be a requirement for any government contracts, so it’s likely in that case. But London? Not so much.
You can’t just trace route this stuff either, since the ISP will have a preferred path which saves them money. That preferred path is likely going to go through the nearest IX regardless if your traffic should head the opposite direction. You can maybe dig through BGP via a looking glass (such as hurricane electrics looking glass at https://lg.he.net ), but that only tells you if the IP range is advertised as accessible from a location, not how it gets from that location to wherever it’s going (IX-IX links only, or is there a geographically diverse path?). It’s an ok indicator, but definitely not a definitive answer. Another good indicator is whether a trace route shows IX related links or not, but without knowledge of whether those links that don’t indicate transit through an IX are actually in an IX/DC connected to an IX or not. It also doesn’t tell you if those are the only links for that path, as the ISP has no responsibility to expose their internal routing information to the public. The IX-IX paths may just be preferred, and the direct paths are simply on standby for them if there’s ever a problem with the inter IX path.
Long story short (and TL;DR): you can get some decent hints but unless you work for the ISP, you can never really know. The non-IX paths don’t necessitate that they are ever used in the internal routing by the ISP, whether they exist or not, and nobody will just go around spilling those beans whenever the question is raised.
P2p mesh is great except there’s no DNS, so now either someone needs to host that, making them a critical node. Either that or every needs to start memorizing IP addresses. Using discovery through broadcast would easily create so much traffic that you’d run out of bandwidth on a wireless mesh.
Everyone seems to think that a wireless mesh is some kind of golden bullet, but it’s really not. Putting that many people on a mesh generates a lot of broadcasts, which by nature go to everyone on the net. Having broadcasts rebroadcast by mesh stations will double the traffic load each time it’s done, so beyond 4-5 mesh points, and a few dozen clients, the broadcasts start to take over the available bandwidth.
Rebuilding, basically means that someone needs to set up an IX, and connect their neighbors to them in an ISP-like configuration. One person basically needs to become the ISP/IX for a neighborhood and connect to others doing the same for their neighborhood.
At least one of the IX owners would need to know enough about networking to set up routing protocols and run an IP address management system to ensure nobody overlaps anyone else, and configure routing protocols for all the IX locations so they can communicate to eachother.
Then there’s the problem of services. The global DNS system is down, so you need to make a new one. Services you expect on the internet are gone, so those need to be rebuilt, which means someone needs to basically become a datacenter to run the servers to generate those services.
I would be the obvious candidate to do this in my area. I know of a handful of people who could do the same in their area, and only one such person lives remotely close to me (about an hour or so drive away). I have my own homelab servers, and more networking equipment than I can shake my fist at. I’d go and raid the local ISP distribution building to pillage some fiber delivery equipment and build out a gigabit+ speed passive optical network from my basement as the ISP/IX/DC.
Finding outdoor rated fiber lines and whatnot to run any relevant connections would be important.
Then set up a wireless point to point link to any other nearby community networks.
A couple of years ago, I was working at a grocery store picking orders using a web app. The store had enterprise-class Internet service with a provider that had two power utility feeds from two different electrical substations a couple miles apart, for reliability.
One day, though, our service went down. One of the power substations had exploded. Shortly, thereafter, the increased load on the grid caused the other substation to explode, too. The cascading electrical failure took out the ISP’s backup generator.
That didn’t even take nuclear war, just a faulty transformer. (ETA: The disaster preparedness lesson is to look for hidden dependencies between your backups.)
The ISP had redundant electrical grid connections for reliability, but the two connections were not isolated at the electrical utility level. A failure in one substation cascaded to the other substation. The operation of one electrical feed depended on the operation of the other, so they were effectively only a single feed.
But I don’t understand why them being connected makes one dependent on the other, unless half of the supply alone can’t support the workload. What is the “electrical utility level”
Both substations feed in parallel. Demand exceeded the capacity of station 2 to supply demand and it tripped the cutout. Only rural connections are radial. Electrical utility starts at the point of demarcation to the generators that make the power.
The “utility level” is Madison Gas & Electric’s infrastructure. Our ISP had two independent electrical service connections based on the idea that if one went down, they’d still get power sufficient to run their data center from the other. That would be the case if each connection reached all the way to the generating station completely independently. However, the two substations to which the ISP was connected were linked in such a way that a catastrophic failure of one caused failure of the other, so it got no electrical power.
I thought the point of the Internet is that in case nuclear apocalypse happened, that there will still be a reliable mode of communication?
Yes and no.
The military internet, which is built on similar principles and protocols, yes. The commercial internet? Ehhh. Not so much.
Most internet interconnects are consolidated into internet exchanges, usually in, or near, datacenters. I live in southern Ontario, pretty much every local ISP routes the majority of their traffic, if not all of it, through TorIX in Toronto. There’s a datacenter there, or more accurately several, which are cross connected into or through TorIX. Many of the local ISPs do not have redundancy with another IX or datacenter. One notable exception is Bell Canada, who has a strong, and broad transportation network of their own, which connects to TorIX and the IX in Montreal, among others. They’re probably the best set up in their distribution layers from what I can tell.
Most ISPs can suffer the loss of one major upstream network provider loss, but not all of them in the IX. Aka, if the datacenter/IX falls, the entire network goes offline.
Take for example a local third party provider of teksavvy. And to be clear, I’m making a lot of wild speculation about teksavvy here, and reality may be very different, but I’m just using them as an example. Normally, small ISPs like teksavvy do one of two things, they either resell DIA connections (direct internet access), which is basically just buying another providers access from them. Aka, you buy teksavvy, but the line you’re connected to is from Bell, your internet and your internet IP is provided by Bell, and you pay teksavvy who pays Bell. You might as well be on Bell, but since teksavvy is buying in bulk, you can usually get service from them cheaper than from Bell. Functionally it’s the same, so there’s a null sum here. The alternative is that teksavvy gets a wholesale connection to you, where Bell (or another “last mile” ISP) provides the connection from your premise to teksavvy, usually through a local IX, your connection still goes into the datacenter, but rather than go from the bell line, through Bell’s gear direct to the internet, it goes from Bell’s wholesale line, to teksavvy gear in the datacenter, then gets rerouted through to the internet from there. All within the DC/IX.
The IX crossconnects all providers in the DC, which would be companies like Meta, Google, Netflix, the company formerly known as Twitter, etc, and also to other ISPs, so you can directly connect to your friend down the road on Rogers without having your traffic go further than it has to. The IX is where ISPs, providers, datacenters and all other connections meet. A nearby nuclear blast that destroys the DC/IX in your area, would very likely disrupt all communication for the area served by ISP connections routed through that IX.
In rare cases (I believe Bell is set up this way), the ISP will cross link it’s distribution centers, which are usually buildings in your neighborhood with the company’s logo, but no customer facing area, to eachother all the way across an area into the next area where the next IX is. So you’re functionally connected to two IX locations or more. The providers distribution network isn’t as fast as the IX to IX direct fiber routes, which usually have fewer devices to go through, but it would work at very limited capacity.
On the public internet, there is a mesh between IX’s, but the location of the IX isn’t hidden. Any individual connection to the internet is usually only connected to a single IX. Most of the distribution between you and the IX is not redundant.
Then compare with what I believe would be a typical example of a military “internet” (again, I’m just speculating based on highly redundant data principles, nothing more): each location, like a military base, airfield, government complex, etc. Would be connected to multiple other locations in a mesh. In addition, they would have backup links likely through satellite or microwave relay links. So if all of the locations that a base is connected to go down, they can use satellite as a final option. Imagine nine such sites in a standard 3x3 grid. Each is connected to no less than 3 neighbors, and the central base is connected to all eight of the others. The central base is destroyed (or otherwise has their datacom disabled). All outlying bases still have at least two links to neighbors. Communication continues. Another base is destroyed, say, the north-East location. All nodes can connect to eachother without issue, but the north and east bases are down to one link plus backup. The South-Eastern location goes down. The rest are unaffected except the eastern base which now needs to rely on satellite backup, but they can communicate. Etc.
As you can hopefully see, the bases all have multiple redundant links and unless they are destroyed or otherwise have their datacom disabled, they can remain in communication with the other bases.
With the internet, that’s usually how IX’s are interconnected, but anything on a single IX, is basically fucked if that IX no longer operates. We’re all nodes of an IX. An intelligent adversary would target any known centers of telecommunications, and we’ve made it easy for them my centralizing all of the communications for a given geography into centralized locations which are conveniently published for anyone to see. There are lists of IX locations on the internet for anyone to stare at, including what ISP companies are connected. With a single well-placed, high yield (not even nuclear) bomb or ICBM, all commercial communication systems can be neutralized for an area of attack. They wouldn’t even need to destroy a full city block to accomplish it.
Sure, bombing, say TorIX, wouldn’t stop someone in, say, Nova Scotia, from chatting to someone in British Columbia, but pretty much everyone in Southern Ontario would be disconnected in an instant. Unable to tweet about the bomb that just went off in downtown Toronto (I believe the IX there is next to Union station, at 151 Front St).
The reason it’s done this way is because of money. It’s far cheaper to centralize access to an IX, then connect the IX’s together. The alternative is to string fiber between datacenter buildings for different companies at different addresses. Getting fiber between geographically different locations is costly, so the companies at the DC/IX all pay a small portion of the fees, either directly or indirectly, to have each IX connected to the others. That cost is shared and the various providers can have access to a limited number of fiber strands that run between locations. Even a handful (like 5) strands can net about 1Tbps of bandwidth or more depending on how it’s used. The planning and deployment of such connections can easily run into the millions of dollars. Sometimes significantly more. Sharing in that cost is a good business move, even if you’re “helping” your competition by doing so, because they also are getting a benefit from it.
Taking it back to the OP: the blast observed appears to be in a city, where it is likely an IX would be situated, most likely the one that your shelter is connected to. Cellular won’t help since a lot of that infrastructure relies on microwave relays back to a head node, usually on the cities outskirts, which then transfers the data to a high-speed fiber line to… You guessed it, the IX.
The internet isn’t designed to be resilient to war. We will lose all datacom if war breaks out, with certainty.
Military networks, though similar in design, have vastly more redundancy, the likes of which, we, the people, do not, and will likely never, have access to.
That was extremely detailed. Thanks. Now I know how to take out the internet.
I skimmed through most of what you said and want to thank you for the knowledge, is there a way to see where a isp has their IX before joining
IX vendor lists are pretty much public information. There’s plenty of ways to find the information but you have to work backwards from how you’re thinking. The way you’ve phrased this is to find which IXes an ISP is connected to, but you can only really find what ISPs are at which IXes.
For example, TorIX, a limited list is literally on their main page of their official site: https://www.torix.ca/
Rinse and repeat for every IX.
If you can find which business segment for an ISP handles consumer connections, you might be able to find them on the peering db ( https://www.peeringdb.com ), which, if you dig hard enough, you could get a list of which IXes the ISP is linked to (aka, peered with). Taking a major ISP here as an example, Rogers, there are eight networks with “Rogers” in the name. A couple are probably not the Rogers ISP/Telco company, but it looks like six might be, each is some business segment. Now, I know that Rogers does most of their internet services using AS 812, so I’d start there, and I can see they’re linked to Toronto, Montreal, Vancouver, new York, Chicago, Seattle, and Ashburn. The question I can’t answer is, does Rogers have a non IX-IX link (something geographically divergent) from, say, Toronto, and their clients there, over to Montreal/new York/Chicago/(etc). Some kind of interconnect that doesn’t require the IX to exist? If I’m in, say, London, Ontario, about two hours west of Toronto, (where my closest IX would be Toronto), and the IX in Toronto goes away, will there be a way for Rogers to get my traffic out to the internet? Or does my traffic go to Toronto and get relayed from there every time?
There’s no possible way to know without having intimate knowledge of the interconnects that Rogers owns and where they are physically run. Even if that fiber link looks like it goes straight from the London area to, say, Chicago, how do you know that the physical fiber doesn’t go towards Toronto, land in the IX then get switched across an IX-IX link, to the Chicago IX? It’s impossible to know.
Turning it around on Montreal, if I’m located in say, Ottawa, which is pretty much midway between Montreal and Toronto (closer to Montreal), and the Montreal IX ceases to exist, will my traffic have a viable path to the Toronto IX? In the same way, I have no idea. It could bounce through Montreal’s IX before going to Toronto over an IX-IX link.
I suspect that the Ottawa situation is much more likely to have dedicated fiber running from there to both Toronto and to Montreal, since it’s the capital of Canada, and having that kind of geographically diverse path to a couple IXes, would be a requirement for any government contracts, so it’s likely in that case. But London? Not so much.
You can’t just trace route this stuff either, since the ISP will have a preferred path which saves them money. That preferred path is likely going to go through the nearest IX regardless if your traffic should head the opposite direction. You can maybe dig through BGP via a looking glass (such as hurricane electrics looking glass at https://lg.he.net ), but that only tells you if the IP range is advertised as accessible from a location, not how it gets from that location to wherever it’s going (IX-IX links only, or is there a geographically diverse path?). It’s an ok indicator, but definitely not a definitive answer. Another good indicator is whether a trace route shows IX related links or not, but without knowledge of whether those links that don’t indicate transit through an IX are actually in an IX/DC connected to an IX or not. It also doesn’t tell you if those are the only links for that path, as the ISP has no responsibility to expose their internal routing information to the public. The IX-IX paths may just be preferred, and the direct paths are simply on standby for them if there’s ever a problem with the inter IX path.
Long story short (and TL;DR): you can get some decent hints but unless you work for the ISP, you can never really know. The non-IX paths don’t necessitate that they are ever used in the internal routing by the ISP, whether they exist or not, and nobody will just go around spilling those beans whenever the question is raised.
deleted by creator
P2p mesh is great except there’s no DNS, so now either someone needs to host that, making them a critical node. Either that or every needs to start memorizing IP addresses. Using discovery through broadcast would easily create so much traffic that you’d run out of bandwidth on a wireless mesh.
Everyone seems to think that a wireless mesh is some kind of golden bullet, but it’s really not. Putting that many people on a mesh generates a lot of broadcasts, which by nature go to everyone on the net. Having broadcasts rebroadcast by mesh stations will double the traffic load each time it’s done, so beyond 4-5 mesh points, and a few dozen clients, the broadcasts start to take over the available bandwidth.
Rebuilding, basically means that someone needs to set up an IX, and connect their neighbors to them in an ISP-like configuration. One person basically needs to become the ISP/IX for a neighborhood and connect to others doing the same for their neighborhood.
At least one of the IX owners would need to know enough about networking to set up routing protocols and run an IP address management system to ensure nobody overlaps anyone else, and configure routing protocols for all the IX locations so they can communicate to eachother.
Then there’s the problem of services. The global DNS system is down, so you need to make a new one. Services you expect on the internet are gone, so those need to be rebuilt, which means someone needs to basically become a datacenter to run the servers to generate those services.
I would be the obvious candidate to do this in my area. I know of a handful of people who could do the same in their area, and only one such person lives remotely close to me (about an hour or so drive away). I have my own homelab servers, and more networking equipment than I can shake my fist at. I’d go and raid the local ISP distribution building to pillage some fiber delivery equipment and build out a gigabit+ speed passive optical network from my basement as the ISP/IX/DC. Finding outdoor rated fiber lines and whatnot to run any relevant connections would be important.
Then set up a wireless point to point link to any other nearby community networks.
deleted by creator
A couple of years ago, I was working at a grocery store picking orders using a web app. The store had enterprise-class Internet service with a provider that had two power utility feeds from two different electrical substations a couple miles apart, for reliability.
One day, though, our service went down. One of the power substations had exploded. Shortly, thereafter, the increased load on the grid caused the other substation to explode, too. The cascading electrical failure took out the ISP’s backup generator.
That didn’t even take nuclear war, just a faulty transformer. (ETA: The disaster preparedness lesson is to look for hidden dependencies between your backups.)
Can you please elaborate on the technical details of the failures? What was the hidden dependency?
The ISP had redundant electrical grid connections for reliability, but the two connections were not isolated at the electrical utility level. A failure in one substation cascaded to the other substation. The operation of one electrical feed depended on the operation of the other, so they were effectively only a single feed.
But I don’t understand why them being connected makes one dependent on the other, unless half of the supply alone can’t support the workload. What is the “electrical utility level”
Both substations feed in parallel. Demand exceeded the capacity of station 2 to supply demand and it tripped the cutout. Only rural connections are radial. Electrical utility starts at the point of demarcation to the generators that make the power.
The “utility level” is Madison Gas & Electric’s infrastructure. Our ISP had two independent electrical service connections based on the idea that if one went down, they’d still get power sufficient to run their data center from the other. That would be the case if each connection reached all the way to the generating station completely independently. However, the two substations to which the ISP was connected were linked in such a way that a catastrophic failure of one caused failure of the other, so it got no electrical power.
Nuclear explosions emit massive EMPs. enough to kill a lot of electronics