The Rogers Outage

[jsburbidge]

 When I work from home, I work online, but I use Bell for internet, Telus for phone, and have an employer who seems not to rely on Rogers at all.[1]. So although I knew that some fellow employees were having to use public WiFi sites because their Rogers connections were down, I gave little thought to the outage until a planned release was deferred on account of the outage, and even then that was because it affected the availability of support staff.  Only when my daughter phoned me to tell me that debit in general was down did I find out that the failure of a single vendor has essentially brought whole blocks of commerce (plus services like 911) to a screeching halt.[2]

Aside from noting that both Rogers and other large services should be looking very carefully at their architectures for redundancy - the easy fix is probably for vendors like Interac which ought to be able to shift to having parallel vendors providing load-balanced access to communications; Ghu knows what Rogers' architecture is like and they are not being very clear - I see that there are calls for steps to be taken to provide more vendors and less dominance by a few vendors. (Essentially two: Bell and Telus share much of the same backbone system.)

This is not a new idea, although usually the reason has been the concern at limited commercial competition, not system reliability. The previous attempts to provide for more vendors have not been successful, at least from the point of view of stability and robustness of the economy as a whole. (The smaller vendors use the large vendors' hardware and rent access in blocks.) This is because the substantial cost of building another backbone is a sizeable barrier to entry.

If the government wants to have another active competitor in the market, it would either have to provide massive subsidies to a startup (this would not fly, politically and perhaps legally) or enter the market itself under a Crown corporation. (Note that the aim of such a corporation would not be to provide monopoly services, as Bell used to or as the LCBO and Ontario Hydro do; that would defeat the purpose. The aim would be to provide more different vendors.) For practical purposes this also means that prices would effectively be set by the government, not just regulated by the CRTC as they are now. (Whatever price was charged by such a Crown corporation would become the de facto ceiling for basic internet services.). It would also see considerable reductions in planned growth for the telcos and probable actual shrinkage of their markets.

Would the mandate of such a company cover all, or most, residents, or would it be confined to the areas more critical to general commerce? Practicality would argue for the former, but politics would probably demand the latter. Costs are higher as a result.

The new system itself would have to provide at every level for a high degree of redundancy and have significant overcapacity in order to handle unexpected eventualities. (Consider an existing vendor choosing to exit the market and its customers moving to the new Crown corporation; and unexpected eventualities are exactly the driving reason for such a system.)

So it would be an expensive, highly contentious, and lengthy initiative which would have to last through multiple governments. (Cheap alternatives like heterodyning IP over the power supply are most useful at the final delivery stage, and do not address the problem of providing for redundancy in the backbone.)

Do I expect this to happen? Not on the basis of a 24-hour incident - though it would be wisest to consider it a shot by the future across our bows.

[1]To the level that it was the only one of the major banks whose debit system was unaffected by the outage. Which didn't help them much, as Interac was affected, which meant that although they were up in principle connections from merchants were down.

[2] I realize that credit was not affected. In some ways that's worse, because the cost of the outage would have fallen disproportionately on the poor, who are less likely to have credit.

Comments

[graydon]

Many things should not be treated as a market; backbone is one of them. I think it's painfully obvious that we can't leave the backbone to the commercial sector.

There are inescapable selection pressures to concentrate like this for a commercial provider, which is the same thing as saying "to break the network".

Backbone should be completely public. Nationalization of existing infrastructure should apply the cost of the outages against the notional value of the infrastructure, and there should be black-letter law SLAs that apply to every Canadian citizen. (It should also destroy Bell, Telus, Rogers, and Shaw root and branch, so that there's zero institutional continuity.)

(One central server setup per not more than a million, at least three failover sites and failover tested monthly at random by a nice deputy minister showing up and cutting the power to a member of a failover network with no warning. Etc. Probably high-altitude solar aircraft in the north, at least for now.)

You get your services from size-limited customer service orgs who all pay the same access prices; you can't buy backbone access unless you're a Canadian organization able to make laws.

It wouldn't be hard to do. I doubt it will happen, alas. The necessity of resilience hasn't made it into politics yet; is being forcibly kept out.

[jsburbidge]

The model to think of is the original Ontario Hydro -- the original Crown Corporation in Ontario - which was established to provide a single reliable generation / distribution system for hydroelectric power (and later other sources of electricity).

Rogers is not only not transparent about this -- I would guess, based on their current statements, that they fixed it by backing out updates but still don't really know what went wrong -- but the last year or two have displayed levels of infighting and simple managerial dysfunction which would make one generally very concerned in any case.

Bell used to be a very tightly-regulated monopoly when they were a POTS company; next door to a Crown corporation but not quite. Their trajectory since the constraints were loosened to both allow for competition and to allow them to spread out into other areas have not been encouraging.

The fundamental idea of packet-switched networks comes from the aim of having a network which is *not* vulnerable to the failure of individual nodes. It takes a certain amount of negative talent to implement it using a "core network" (Rogers term) which subverts the whole purpose.