US - Canada blackout report

[Richard]

https://reports.energy.gov/

I'm not an electrical engineeer, but, makes interesting reading.

 

[Charles Perry]

https://reports.energy.gov/

I'm not an electrical engineeer, but, makes interesting reading.

I am an electrical engineer...in the power industry. I read the entire
report last night and found it very interesting. I am dissappointed (but
not surprised) that they did not include any discussion on how the
deregulation of the power industry has helped create the situation that led
to the blackout. Of course I never expected FERC to point a finger at
itself.

Charles Perry P.E.

 

[anon]

I have not read the complete report yet.
So far I have not seen any comments about the level of staffing.
Staffing has changed substantially due in part to de regulation.

former electrical operator

 

[Don Kelly]

Sorry, but you are missing the point. A system that can take 5% of
the world economy off line is not acceptable (regardless of the
blaming or chaotic stimulus). There should not be a large grid. This
was created to allow sharing. The failure domain should be smaller!

This can only be achieved by breaking the grid into smaller domains.
This will limit selling of power from state to state, but insure that
OH can't take the east coast down.

Solve the problem, don't play games blaming nature or management.
If all the planes fell out of the sky based on one person's failure,
no one would fly. Failure group size is too large, divide the system.

Rob

---------------
Actually, a well run large grid will be more economical and also more
reliable than a bunch of small ones. In the case of smaller, isolated
systems, the impact would be greater, in the form of a thousand small cuts
which would bleed the economy even more. Note: "well run" does not mean
maximising profits. It does mean that the people at the top should
understand power systems and not have "business "experience that considers
utilities, grocery stores, department stores, etc as interchangable.
Deregulation, has, in many places, put short term profits ahead of long term
planning for reliability and adequacy of supply.

 

[John Gilmer]

Sorry, but you are missing the point. A system that can take 5% of
the world economy off line is not acceptable (regardless of the
blaming or chaotic stimulus). There should not be a large grid. This
was created to allow sharing. The failure domain should be smaller!

Nonsense!

MOST of the time, the large grid (aside from "wheeling" excess power to
where is it needed from where it is not) increases reliability. This is
expecially true for customers at the "edge" of a service area.

This effect is so important that utilities at the "edge" of the major grids
often establish DC links to the adjacent grid.

This can only be achieved by breaking the grid into smaller domains.
This will limit selling of power from state to state, but insure that
OH can't take the east coast down.

Well, the "east coast" (actually only a part of it) went down because the
local utilities were unable to maintain operations when the grid had
problems. Each utility has a obligation to its customers to maintain the
ability (within its own resources) to determine if a major problem with the
"grid" exists and temporarily and without damage to its backbone
distribution system and power plants shed or adjust load and production as
necessary.

IOW: The systems in NY and Canader should have been able to get back on
line for most of its customers within minutes.

NYC, for example, was not even able to restore power to its traffic signals.
It's irresponsible to blame this failure on a utility in the midwest.

 

[tom smith]

You continue to use terms like "well run" and "people should understand".
They aren't and they don't. Good system design takes the fool into account.

You can't claim "it was a great design except the user screwed it up" -- It
was a bad design.

The root cause was not just a failed alarm, it was a system design that let
a failed alarm create
a massive catastrophic failure that in many cases took over a week to bring
back up.

This is an engineering group is it not? Let's face the facts: BAD DESIGN!!

Rob

I agree that the power traders stole money from us after deregulation. They
squeezed money
from a system that we all paid for and failed to invest in it. Now they
want to raise our rates to
fix it.

 

[daestrom]

Nonsense!

MOST of the time, the large grid (aside from "wheeling" excess power to
where is it needed from where it is not) increases reliability. This is
expecially true for customers at the "edge" of a service area.

This effect is so important that utilities at the "edge" of the major grids
often establish DC links to the adjacent grid.
Agreed.


Well, the "east coast" (actually only a part of it) went down because the
local utilities were unable to maintain operations when the grid had
problems. Each utility has a obligation to its customers to maintain the
ability (within its own resources) to determine if a major problem with the
"grid" exists and temporarily and without damage to its backbone
distribution system and power plants shed or adjust load and production as
necessary.

This is not that simple. The CNY area *had* sufficient local power
available to sustain itself. In fact, some portions of CNY didn't even go
dark. With some of the 'islands' that resulted, we were able to bring
several units on-line without having to wait for power from outside our
region.

IOW: The systems in NY and Canader should have been able to get back on
line for most of its customers within minutes.

This is not true. To think that a plant that is tripped off-line can be
restored in minutes shows a lack of understanding of plant operations. Many
plants are not even capable of doing a 'black' startup. The cost of such a
capability is high, and the need for it is *very* rare. Most plants are
still recovering from the trip for some time, they can't even begin to think
about coming back on line 'within minutes', they are still stabilizing from
the trip.

NYC, for example, was not even able to restore power to its traffic signals.
It's irresponsible to blame this failure on a utility in the midwest.

A real issue is that plant trips occurred with a transient that did not
isolate regions. Seems like the line settings should have isolated regions
*before* plants begin to trip. So a severe transient would isolate regions,
not cause a wide-spread generation trip. But the 'isolation zones' have to
be large enough to include a number of base-load *and* regulating units so
that each region can reasonably survive such an 'isolation event'.

There is obviously a tradeoff here. A larger region is more likely to
survive isolation provided it encompasses enough regulating units, spinning
reserves, MVAR capacity and base load. The region has to be able to survive
when some plants (including regulating) are down for maintenance. Today,
the utility saves the ratepayer money by 'borrowing' the services of
regulating and base load from nearby regions through interties. If not for
this ability to rely on neighboring regions for such services, each region
would have to overbuild equipment that would be underutilized except for the
'once every 30 years transient isolation event'.

But wait, we've just 'designed' a system remarkably similar to today's.
Moral is, if you want more reliability than we have, build more, pay more.
Granted, the 8/14 blackout is still 'fresh' in everyone's mind, but is it
*really* a warning of an unreliable system, signaling a future of frequent
blackouts?
------
A second issue is ratepayers paying for equipment upgrades so that more
power can be carried *through* their region. Should ratepayers in rural NY
pay to upgrade equipment such that customers in NYC can buy cheap power from
the Midwest? The large amount of power flow *through* a region is a burden
on the equipment, while the local users see little benefit.

Restructuring 'carrying charges' so that the NYC consumer pays a more
realistic fee for the use of the equipment used to transmit the power from
the Midwest through the rural NY utility seems in order. Then the rural
utility can justify equipment upgrades without burdening the local ratepayer
that receives little from such an upgrade. Rural NY is facing rate
increases so that NYC and mid-west can buy/sell more power from each other.
Louisiana is in a similar situation between Texas and Florida. They are
being asked to upgrade their east-west transmission capabilities at the
expense of LA ratepayers, so Texas and Florida can trade more power.

-----
Thirdly, some consider our limited transmission capability as our only way
to keep Midwest coal plants from making more acid rain for NY and east. If
older OH coal generation units (exempt from anti-pollution upgrades) are
given more access to distant markets, they may operate full time,
exacerbating the acid rain problems in NY and New England.

daestrom

 

[Don Kelly]

You continue to use terms like "well run" and "people should understand".
They aren't and they don't. Good system design takes the fool into account.

You can't claim "it was a great design except the user screwed it up" -- It
was a bad design.

The root cause was not just a failed alarm, it was a system design that let
a failed alarm create
a massive catastrophic failure that in many cases took over a week to bring
back up.

This is an engineering group is it not? Let's face the facts: BAD DESIGN!!

Rob

I agree that the power traders stole money from us after deregulation. They
squeezed money
from a system that we all paid for and failed to invest in it. Now they
want to raise our rates to
fix it.

________
If you had read my comments -I am not claiming the system was well run
although parts of it were. . Good system design is foolproof but not
damfoolproof, particularly where the top management falls into the latter
category and good design is compromised because decisions are made by those
whose only concern is the immediate bottom line and engineers are just hired
help who "don't see the big picture".

 

[John Gilmer]

This is not that simple. The CNY area *had* sufficient local power
available to sustain itself. In fact, some portions of CNY didn't even go
dark. With some of the 'islands' that resulted, we were able to bring
several units on-line without having to wait for power from outside our
region.


This is not true. To think that a plant that is tripped off-line can be
restored in minutes shows a lack of understanding of plant operations.

You have the advantage over me in that I have never participated in day to
day power plant management.

I do, nontheless, understand that a "Panic Button" shut down (with complete
shutdown, steam dumping, turbines coasting to a stop, etc,) might well
require inspections and a SLOW re-start.

Many
plants are not even capable of doing a 'black' startup. The cost of such a
capability is high, and the need for it is *very* rare.

It is my understanding that the FEDs require that nuke plants have enough
diesel generator capacity to do a "black" startup. (Even if it isn't a
formal requirement, safety requirements providing for operation of safety
related sysems (in a nuke plant, that covers a lot of territory).

Most plants are
still recovering from the trip for some time, they can't even begin to think
about coming back on line 'within minutes', they are still stabilizing from
the trip.


A real issue is that plant trips occurred with a transient that did not
isolate regions. Seems like the line settings should have isolated regions
*before* plants begin to trip. So a severe transient would isolate regions,
not cause a wide-spread generation trip. But the 'isolation zones' have to
be large enough to include a number of base-load *and* regulating units so
that each region can reasonably survive such an 'isolation event'.

There is obviously a tradeoff here. A larger region is more likely to
survive isolation provided it encompasses enough regulating units, spinning
reserves, MVAR capacity and base load. The region has to be able to survive
when some plants (including regulating) are down for maintenance. Today,
the utility saves the ratepayer money by 'borrowing' the services of
regulating and base load from nearby regions through interties. If not for
this ability to rely on neighboring regions for such services, each region
would have to overbuild equipment that would be underutilized except for the
'once every 30 years transient isolation event'.

But wait, we've just 'designed' a system remarkably similar to today's.
Moral is, if you want more reliability than we have, build more, pay more.

Well, nothing is "free." But that's not the same a throwing money at the
problem as was proposed in the weeks following the blackout.

Granted, the 8/14 blackout is still 'fresh' in everyone's mind, but is it
*really* a warning of an unreliable system, signaling a future of frequent
blackouts?
------
A second issue is ratepayers paying for equipment upgrades so that more
power can be carried *through* their region. Should ratepayers in rural NY
pay to upgrade equipment such that customers in NYC can buy cheap power from
the Midwest? The large amount of power flow *through* a region is a burden
on the equipment, while the local users see little benefit.

Again, you have the advantage but it seems to me that you are talking about
a variation of the "Interstate Highway Problem" whereby the interstates
become the main rural routes and all the stores and new construction happens
within a few miles of interstate exits.

A rural utility has the option of builing lines to its neighbors on its own
or just using a line constructed (say) to bring power from Niagra Falls to
NYC. Maybe things are different in NYS, but in Virginia rural folks pay a
fraction of the rates that suburban or urban folks pay.

Restructuring 'carrying charges' so that the NYC consumer pays a more
realistic fee for the use of the equipment used to transmit the power from
the Midwest through the rural NY utility seems in order. Then the rural
utility can justify equipment upgrades without burdening the local ratepayer
that receives little from such an upgrade. Rural NY is facing rate
increases so that NYC and mid-west can buy/sell more power from each

other.

Well, I would be interesting a hearing about specific examples.

Louisiana is in a similar situation between Texas and Florida. They are
being asked to upgrade their east-west transmission capabilities at the
expense of LA ratepayers, so Texas and Florida can trade more power.

Well, don't the owners of transmission lines get paid for carrying power?

-----
Thirdly, some consider our limited transmission capability as our only way
to keep Midwest coal plants from making more acid rain for NY and east. If
older OH coal generation units (exempt from anti-pollution upgrades) are
given more access to distant markets, they may operate full time,
exacerbating the acid rain problems in NY and New England.

Interesting issue.

I recall reading that there is a question as to whether "improvements" to
existing coal fired plants should require that the entire plant meet
"modern" anti-polution standards.

Maybe the best solution is a "polution" tax. But the utilities don't like
the tax and the "enviromental wackos" do like that idea of "approved"
polution.

 

[Don Kelly]

I am an electrical engineer...in the power industry. I read the entire
report last night and found it very interesting. I am dissappointed (but
not surprised) that they did not include any discussion on how the
deregulation of the power industry has helped create the situation that led
to the blackout. Of course I never expected FERC to point a finger at
itself.

Charles Perry P.E.

 

[anon]

One of the major issues an operator always faces is 100 percent hindsight in
a quiet back office a week, a month or a year later.

"operators who did not think" ? The alarm system had failed! What actual
issues/events/communications were occurring in the control room
at the time of the alarm system failure?

From Computerworld (November 24) - as taken from the task force report

"... a software program that gives operators visual and audible indications
of events occurring on their portion of the grid, began to malfunction.
As a result, ' key personnel may not have been aware of the need to take
preventive measures ..."
....

The statement below indicates the system board that was promised in planning
the control room design was deleted to save $'s.
Definitely a typical management shortcoming taking away the operators tools.

'this may have been, in part, the result of a failure to use modern dynamic
mapping and data sharing systems'

The operators were always convenient targets.

Re failure to trim trees - are there any forestry personnel who are utility
staff (not contractors) left at the utilities?

former Electrical Operator

 

[daestrom]

I do, nontheless, understand that a "Panic Button" shut down (with complete
shutdown, steam dumping, turbines coasting to a stop, etc,) might well
require inspections and a SLOW re-start.

Exactly. Returning a unit to service can take quite some time. Tripping
the boiler off and blowing the safety's for several minutes is pretty
severe. Getting the turbine back is also a slow process. Massive machinery
requires one to adhere to heat-up /cool-down limitations to avoid damage.
And of course, you have to have a 'grid' to synch back on to unless you are
a regulating unit.

It is my understanding that the FEDs require that nuke plants have enough
diesel generator capacity to do a "black" startup. (Even if it isn't a
formal requirement, safety requirements providing for operation of safety
related sysems (in a nuke plant, that covers a lot of territory).

Very WRONG!! The NRC requires taht nuke plants have emergency generators to
put the plant in a safe condition and maintain it in that safe condition.
That is to say, the diesels are used to provide power necessary for cooling
and monitoring the reactor, and ensuring the safety of the reactor. This is
a far cry from that needed to perform a startup.

Again, you have the advantage but it seems to me that you are talking about
a variation of the "Interstate Highway Problem" whereby the interstates
become the main rural routes and all the stores and new construction happens
within a few miles of interstate exits.

A rural utility has the option of builing lines to its neighbors on its own
or just using a line constructed (say) to bring power from Niagra Falls to
NYC. Maybe things are different in NYS, but in Virginia rural folks pay a
fraction of the rates that suburban or urban folks pay.

The question is one of simply "Who pays for the transmission line, vs. who
benefits from its construction?" If the rural community 'pays' but recieves
no benefit, while the city 'benefits' but does not pay, then we have a
problem. With the interstates, everyone 'paid' while not everyone benefited
equally. But imagine if only the rural community paid and the urban folks
did not?

When crossing territories and state lines, this *could* be the result.
Users in FL getting the benefit of larger transmission across LA from TX
(mandated by congress), yet only the LA ratepayers footing the bill?

Well, don't the owners of transmission lines get paid for carrying power?


Interesting issue.

I recall reading that there is a question as to whether "improvements" to
existing coal fired plants should require that the entire plant meet
"modern" anti-polution standards.

Very true. The lawyers/lawmakers get to argue about just what constitutes
an 'improvement'. Replace an old worn out motor/pump for the condensate
system with a newer one. Substantially the same overall, but made with
newer materials, lower maintenance shaft-seals, higher-quality,
longer-lasting bearings. Lowers maintenance costs, replaces an obsolete
component, but is it an 'improvement' that warrants upgrading the pollution
equipment? Probably not, at least the owners think so.

But replace the turbine rotor with a new one. Old rotors *do* wear out, and
erode. New one made from better materials, less bypass losses, more MWe
output. Is *this* an improvement? Owners claim 'like component
replacement', but opponents argue 'improved plant output'.

Maybe the best solution is a "polution" tax. But the utilities don't like
the tax and the "enviromental wackos" do like that idea of "approved"
polution.

As with many things, compromise will probably rule.

daestrom