jsburbidge (![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png)
jsburbidge) wrote2022-07-09 11:17 am
jsburbidge) wrote2022-07-09 11:17 am
Entry tags:
The Rogers Outage
When I work from home, I work online, but I use Bell for internet, Telus for phone, and have an employer who seems not to rely on Rogers at all.[1]. So although I knew that some fellow employees were having to use public WiFi sites because their Rogers connections were down, I gave little thought to the outage until a planned release was deferred on account of the outage, and even then that was because it affected the availability of support staff. Only when my daughter phoned me to tell me that debit in general was down did I find out that the failure of a single vendor has essentially brought whole blocks of commerce (plus services like 911) to a screeching halt.[2]
Aside from noting that both Rogers and other large services should be looking very carefully at their architectures for redundancy - the easy fix is probably for vendors like Interac which ought to be able to shift to having parallel vendors providing load-balanced access to communications; Ghu knows what Rogers' architecture is like and they are not being very clear - I see that there are calls for steps to be taken to provide more vendors and less dominance by a few vendors. (Essentially two: Bell and Telus share much of the same backbone system.)
This is not a new idea, although usually the reason has been the concern at limited commercial competition, not system reliability. The previous attempts to provide for more vendors have not been successful, at least from the point of view of stability and robustness of the economy as a whole. (The smaller vendors use the large vendors' hardware and rent access in blocks.) This is because the substantial cost of building another backbone is a sizeable barrier to entry.
If the government wants to have another active competitor in the market, it would either have to provide massive subsidies to a startup (this would not fly, politically and perhaps legally) or enter the market itself under a Crown corporation. (Note that the aim of such a corporation would not be to provide monopoly services, as Bell used to or as the LCBO and Ontario Hydro do; that would defeat the purpose. The aim would be to provide more different vendors.) For practical purposes this also means that prices would effectively be set by the government, not just regulated by the CRTC as they are now. (Whatever price was charged by such a Crown corporation would become the de facto ceiling for basic internet services.). It would also see considerable reductions in planned growth for the telcos and probable actual shrinkage of their markets.
Would the mandate of such a company cover all, or most, residents, or would it be confined to the areas more critical to general commerce? Practicality would argue for the former, but politics would probably demand the latter. Costs are higher as a result.
The new system itself would have to provide at every level for a high degree of redundancy and have significant overcapacity in order to handle unexpected eventualities. (Consider an existing vendor choosing to exit the market and its customers moving to the new Crown corporation; and unexpected eventualities are exactly the driving reason for such a system.)
So it would be an expensive, highly contentious, and lengthy initiative which would have to last through multiple governments. (Cheap alternatives like heterodyning IP over the power supply are most useful at the final delivery stage, and do not address the problem of providing for redundancy in the backbone.)
Do I expect this to happen? Not on the basis of a 24-hour incident - though it would be wisest to consider it a shot by the future across our bows.
[1]To the level that it was the only one of the major banks whose debit system was unaffected by the outage. Which didn't help them much, as Interac was affected, which meant that although they were up in principle connections from merchants were down.
[2] I realize that credit was not affected. In some ways that's worse, because the cost of the outage would have fallen disproportionately on the poor, who are less likely to have credit.
no subject
Many things should not be treated as a market; backbone is one of them. I think it's painfully obvious that we can't leave the backbone to the commercial sector.
There are inescapable selection pressures to concentrate like this for a commercial provider, which is the same thing as saying "to break the network".
Backbone should be completely public. Nationalization of existing infrastructure should apply the cost of the outages against the notional value of the infrastructure, and there should be black-letter law SLAs that apply to every Canadian citizen. (It should also destroy Bell, Telus, Rogers, and Shaw root and branch, so that there's zero institutional continuity.)
(One central server setup per not more than a million, at least three failover sites and failover tested monthly at random by a nice deputy minister showing up and cutting the power to a member of a failover network with no warning. Etc. Probably high-altitude solar aircraft in the north, at least for now.)
You get your services from size-limited customer service orgs who all pay the same access prices; you can't buy backbone access unless you're a Canadian organization able to make laws.
It wouldn't be hard to do. I doubt it will happen, alas. The necessity of resilience hasn't made it into politics yet; is being forcibly kept out.
no subject
no subject
I would prefer a setup where there's in effect a size cap on core service provision nodes; one per million people.
Nodes are competitively ranked for reliability and value delivered (as a means of controlling the pay and promotions of the people in charge of them) and more or less forbidden from using the same hardware or software as their failover nodes. (more or less = unrelated linux distros probably OK, linux versus BSD definitely OK, all versions of IBM mainframes are the same thing, etc. It would take some taxonomic effort.)
Every failover connection uses different cable runs, in the sense of being physically separate. I don't think we could argue for keeping some copper around but presumably there's alternatives for types of fibre depending on range.
All the nodes have to present the same access protocols, though; the layer of actual ISPs, the people you or I would contract with to get internet, all use the same interfaces on both sides. Hopefully this makes switching easy; hopefully it allows ISPs to have buddy agreements where is the ISP's hardware fails they can fail over to an adjacent ISPs and the customers don't notice.
Northern regions might be a special case, but the likely price point for long-duration solar high-altitude loitering aircraft looks low enough to fly redundant backups. (And to have some surge capacity on hand in adjacent regions; those aircraft aren't fast but have approximately infinite range once they're up there.)
no subject
no subject
Some provinces need more than one!
Though this is definitely an area where I want it to stay in the federal purview.
no subject
no subject
We have crossed swords in the learned Mr. Burbidge’s comments section before. As I recall, you wanted the government to ban advertising, and other speech by Nazis and anyone else “having an attack of supremacy”; my view was and is that, aside from other objections, we cannot trust the government with this kind of power, which might end up being used in ways you really would not like.
no subject
You think the existing backbone doesn't suppress comms? Look into what happens to people trying to complain about Bell charities as one example. (Or how incredibly difficult it is to obtain an effective ad blocker.)
Nothing is safe. Nothing is ever safe, or can be safe. Safe is one of those delusions borne of a capacity to hypothesize.
Power does not stop existing because of how you feel about it. Lots of it currently exists, and is exercised to purpose inimical to anyone who doesn't have it, which is statistically everyone.
What Rogers (and the other Canadian telcos, though Rogers has the worst corporate culture and is ahead of the curve) does is bill you, and increase the amount that it bills you. All else is entirely secondary and has been for decades. Yet Rogers is now making decisions about the entire Canadian economy with no accountability whatsoever.
We are moving into a time when field agriculture fails; we are moving into a time when most of humanity will be displaced from their homes. (This is not grounds to suppose late capitalism is a defensible system! It's put a lot of effort into prevent the public consensus that these outcomes are bad from becoming operant.) There are going to be a great many things I don't like even before I die. "I won't like that" is a silly thing to be worried about.
The system needs to function; the civil power least retains a de jure notion that "function" is something individual citizens get to have an operant opinion about. It can be designed to regard feedback.
"Trust" is a scale error, like morals; of course you should not ever trust anything on larger-than-immediate-personal scales. You should be looking for sufficient feedback, and aware that you might not be able to get it, or get it in the form you most want. Politics-of-feels produces horrors; politics of functional specification and measured results can work OK. It's not guaranteed, but it's possible, as it is not possible of the feels. (There isn't enough information in the feels to control a sufficiently complex system.)
no subject
Efficient is quite likely with a Canadian government bureaucracy along the appropriate measure -- service returned for dollars in.
Innovative in terms of mechanism is NOT desired; resiliency is desired. Innovation in resiliency, if that's where the feedback goes -- you measure uptime -- is quite likely. "Innovation" is generally the problem with the network; it's been innovated into an unreliable condition.
Competent is a function of the measures applied, and very often achieved.
no subject
Rogers is not only not transparent about this -- I would guess, based on their current statements, that they fixed it by backing out updates but still don't really know what went wrong -- but the last year or two have displayed levels of infighting and simple managerial dysfunction which would make one generally very concerned in any case.
Bell used to be a very tightly-regulated monopoly when they were a POTS company; next door to a Crown corporation but not quite. Their trajectory since the constraints were loosened to both allow for competition and to allow them to spread out into other areas have not been encouraging.
The fundamental idea of packet-switched networks comes from the aim of having a network which is *not* vulnerable to the failure of individual nodes. It takes a certain amount of negative talent to implement it using a "core network" (Rogers term) which subverts the whole purpose.
Rogers and CRTC