News

Dispatcher error triggered Tuesday power outage in Palo Alto

City ordered to cut power after dispatcher at Northern California Power Agency misunderstood directive from grid operator

Transformers at the East Meadow substation at the intersection of East Meadow Drive and Middlefield Road in Palo Alto on Jan. 31, 2022. Photo by Magali Gauther.

When the city of Palo Alto shut off power to about 1,700 customers in the Midtown, Old Palo Alto and Industrial Park neighborhoods on Tuesday afternoon, utilities officials believed they were following an urgent order to conserve power from a state agency that oversees independent utility operators.

Now, however, it appears that the power outage was premature, unnecessary and based on a dispatcher's misunderstanding. The outage, which came in the midst of a sweltering heat wave, hit Palo Alto customers at about 6:30 p.m. and lasted for about 30 minutes before city officials learned about the error and restored power.

Palo Alto was asked to cut off power by the Northern California Power Agency (NCPA), a not-for-profit organization that represents numerous municipal utilities and that serves as an intermediary between these utilities and the California Independent System Operator (CAISO), which manages electricity flow. At about 5:17 p.m., CAISO issued an order known as an "energy emergency alert 3 (EE3)," a step that indicates that power outages may need to be implemented.

In receiving the order, however, a dispatcher with the Northern California Power Agency misinterpreted it as a request to shed 46.02 megawatts of electricity to prevent widespread outages, according to a statement from the agency. The dispatcher then immediately communicated the directive to utility officials in Palo Alto, Alameda, Lodi, Santa Clara, Healdsburg and Ukiah, prompting them to turn off power.

Once the dispatcher contacted CAISO to inform them about the action, the individual was notified that there was a misunderstanding around the order. The NCPA then began the process of returning the load back into the system, according to its statement. The longest outages lasted about an hour, the NCPA reported.

Help sustain the local news you depend on.

Your contribution matters. Become a member today.

Join

In explaining the error, the NCPA noted that the issuance of the order represented the first time such an order had been declared in California. It describes a situation in which CAISO is "unable to meet minimum requirements and controlled power curtailments are imminent and in progress." The NCPA stated that it is conducting an internal inquiry into the disconnect and has taken "immediate measures" to make sure all protocols are clear to all involved.

It is also "closely coordinating with the CAISO to review the conversation that occurred between the dispatchers involved to avoid a miscommunication in the future."

NCPA General Manager Randy Howard said in a statement that the outage, while "very impactful to those directly affected," may have helped the state's stressed grid.

"The action taken yesterday by the NCPA member systems mentioned above, while inadvertent, helped to ultimately benefit and keep the overall grid stable through the most extreme heat of the day and a record CAISO load," NCPA General Manager Randy Howard said in a statement Wednesday.

Palo Alto received instructions to begin implementing rolling blackouts at about 5:45 p.m., according to an update that the city posted on its website Wednesday. The rolling blackouts were implemented "in a specific order to minimize customer impact," the city's announcement stated.

Stay informed

Get daily headlines sent straight to your inbox in our Express newsletter.

Stay informed

Get daily headlines sent straight to your inbox in our Express newsletter.

"We share NCPA's regret that this impacted our community and appreciate the patience and support provided through this challenging time," City Manager Ed Shikada said in a statement. "The City is continuing to work on minimizing customer impacts going forward."

The Tuesday outage was one of three that occurred in Palo Alto during this week's heat wave. On Monday night, more than 4,400 customers in south Palo Alto lost power for several hours because of a faulty transformer. On Wednesday, about the same number of customers in downtown Palo Alto lost power for about two hours after a squirrel made contact with an underground wire, according to utilities officials.

Gennady Sheyner
 
Gennady Sheyner covers the City Hall beat in Palo Alto as well as regional politics, with a special focus on housing and transportation. Before joining the Palo Alto Weekly/PaloAltoOnline.com in 2008, he covered breaking news and local politics for the Waterbury Republican-American, a daily newspaper in Connecticut. Read more >>

Follow Palo Alto Online and the Palo Alto Weekly on Twitter @paloaltoweekly, Facebook and on Instagram @paloaltoonline for breaking news, local events, photos, videos and more.

Stay informed on important city government news. Sign up for our FREE daily Express newsletter.

Dispatcher error triggered Tuesday power outage in Palo Alto

City ordered to cut power after dispatcher at Northern California Power Agency misunderstood directive from grid operator

by / Palo Alto Weekly

Uploaded: Thu, Sep 8, 2022, 1:17 am

When the city of Palo Alto shut off power to about 1,700 customers in the Midtown, Old Palo Alto and Industrial Park neighborhoods on Tuesday afternoon, utilities officials believed they were following an urgent order to conserve power from a state agency that oversees independent utility operators.

Now, however, it appears that the power outage was premature, unnecessary and based on a dispatcher's misunderstanding. The outage, which came in the midst of a sweltering heat wave, hit Palo Alto customers at about 6:30 p.m. and lasted for about 30 minutes before city officials learned about the error and restored power.

Palo Alto was asked to cut off power by the Northern California Power Agency (NCPA), a not-for-profit organization that represents numerous municipal utilities and that serves as an intermediary between these utilities and the California Independent System Operator (CAISO), which manages electricity flow. At about 5:17 p.m., CAISO issued an order known as an "energy emergency alert 3 (EE3)," a step that indicates that power outages may need to be implemented.

In receiving the order, however, a dispatcher with the Northern California Power Agency misinterpreted it as a request to shed 46.02 megawatts of electricity to prevent widespread outages, according to a statement from the agency. The dispatcher then immediately communicated the directive to utility officials in Palo Alto, Alameda, Lodi, Santa Clara, Healdsburg and Ukiah, prompting them to turn off power.

Once the dispatcher contacted CAISO to inform them about the action, the individual was notified that there was a misunderstanding around the order. The NCPA then began the process of returning the load back into the system, according to its statement. The longest outages lasted about an hour, the NCPA reported.

In explaining the error, the NCPA noted that the issuance of the order represented the first time such an order had been declared in California. It describes a situation in which CAISO is "unable to meet minimum requirements and controlled power curtailments are imminent and in progress." The NCPA stated that it is conducting an internal inquiry into the disconnect and has taken "immediate measures" to make sure all protocols are clear to all involved.

It is also "closely coordinating with the CAISO to review the conversation that occurred between the dispatchers involved to avoid a miscommunication in the future."

NCPA General Manager Randy Howard said in a statement that the outage, while "very impactful to those directly affected," may have helped the state's stressed grid.

"The action taken yesterday by the NCPA member systems mentioned above, while inadvertent, helped to ultimately benefit and keep the overall grid stable through the most extreme heat of the day and a record CAISO load," NCPA General Manager Randy Howard said in a statement Wednesday.

Palo Alto received instructions to begin implementing rolling blackouts at about 5:45 p.m., according to an update that the city posted on its website Wednesday. The rolling blackouts were implemented "in a specific order to minimize customer impact," the city's announcement stated.

"We share NCPA's regret that this impacted our community and appreciate the patience and support provided through this challenging time," City Manager Ed Shikada said in a statement. "The City is continuing to work on minimizing customer impacts going forward."

The Tuesday outage was one of three that occurred in Palo Alto during this week's heat wave. On Monday night, more than 4,400 customers in south Palo Alto lost power for several hours because of a faulty transformer. On Wednesday, about the same number of customers in downtown Palo Alto lost power for about two hours after a squirrel made contact with an underground wire, according to utilities officials.

Comments

Bystander
Registered user
Another Palo Alto neighborhood
on Sep 8, 2022 at 7:23 am
Bystander, Another Palo Alto neighborhood
Registered user
on Sep 8, 2022 at 7:23 am

I suppose this means that we can't blame Palo Alto Utilities for this outage. However, 3 outages in 3 days seems a lot.


South PA resident
Registered user
Midtown
on Sep 8, 2022 at 9:44 am
South PA resident , Midtown
Registered user
on Sep 8, 2022 at 9:44 am

Glad our PA city council is so focused on mandating electric stoves and hot water heaters so we can lose the use of those too during regular blackouts.

How bout you work on fixing underlying problems before virtue signaling on climate change?


Carl Jones
Registered user
Palo Verde
on Sep 8, 2022 at 10:49 am
Carl Jones, Palo Verde
Registered user
on Sep 8, 2022 at 10:49 am

These are new and different times. As individuals and families, we should come to expect (or at least be prepared for) the previously unexpected.


jguislin
Registered user
Crescent Park
on Sep 8, 2022 at 11:21 am
jguislin, Crescent Park
Registered user
on Sep 8, 2022 at 11:21 am

The very least that PA Utilities owes its customers is to explain how the miscommunication happened and what steps they are taking to ensure it does not repeat. I am less interested in blame and more interested in solutions that make our power more reliable.


Robert
Registered user
Midtown
on Sep 8, 2022 at 11:35 am
Robert , Midtown
Registered user
on Sep 8, 2022 at 11:35 am

So the powers to be decided to shut down the Midtown area because it was the least disruptive? I think the business and homeowners would differ with that opinion its obvious the city doesn't care about South Palo Alto.


Bert
Registered user
Green Acres
on Sep 8, 2022 at 12:35 pm
Bert, Green Acres
Registered user
on Sep 8, 2022 at 12:35 pm

Why is the PA weekly still saying that Wednesday morning's outage was in the downtown area? It was again in the southern part of Palo Alto. Even the graphic from the utility dept's outage map in the original article clearly shows the outage was not downtown. Doesn't the exact same number of customers impacted Monday night and Wednesday morning trigger some suspicions (and the fact that multiple readers have already commented on the actual location of Wednesday morning's outage)?


Annette
Registered user
College Terrace
on Sep 8, 2022 at 1:17 pm
Annette, College Terrace
Registered user
on Sep 8, 2022 at 1:17 pm

What does “after a squirrel made contact with an underground wire” mean? Are the cables not shielded? Is there a way to identify where this vulnerability exists so that it can be remedied? It is a big surprise to learn that power can be lost because of contact with a squirrel. Surely this is fixable. City Manager, what can you say about this?


William Hitchens
Registered user
Mountain View
on Sep 8, 2022 at 5:45 pm
William Hitchens, Mountain View
Registered user
on Sep 8, 2022 at 5:45 pm

NCPA General Manager Randy Howard (I think it was he), appeared live on Channel 2 morning News, on Sunday I think, and he gave a passionate and extraordinary plea for people to conserve additional power between 3 PM and 10 PM --- expanded from the previous norm of 4 PM to 9 PM. He was VERY CONCERNED about CA power demand exceeding available power supply during the present heat wave, which has been worse than projected on Sunday.

I've worked closely a lot with VERY competent engineers like him during my career as an applied physicist, I respect them highly, and his unspoken message was very clear to me. The power grid in CA is very over-stressed and vulnerable, and he feared widespread and uncontrolled blackouts as the grid failed due to lack of capacity and equipment failures due to the heat.

After his words, I immediately contacted Generac and applied for installation of a 20KW natural gas powered backup whole house generator at our house. Why? Things are just going to get worse in a hurry because CA and cities like Palo Alto are doing NOTHING to make their power supplies abundant and robust. Guess they're not power engineers, or in my case, a military industrial complex physicist.

Technically ignorant politicians and activists are forcing people to switch to "green solar and wind power", which are inherently unreliable 24/7/365. This is all well and good, as long as there are backup, modern and efficient natural gas power plants to take over during "green power shortages". BUT guess what? Those same ignorant politicians and ignorant activists are blocking attempts to provide enough NG-fired backup plants, as well as to increase the capacity of CA high tension lines, substations, and the capacity of local power grids.

So things will get a lot worse. That's why I'm installing a backup whole house NG generator. If fools won't take care of me and my family, then I'll take care of myself. To "heck" with the idiot dreamers.


Mondoman
Registered user
Green Acres
on Sep 8, 2022 at 10:28 pm
Mondoman, Green Acres
Registered user
on Sep 8, 2022 at 10:28 pm

@William
I think many would be interested to hear about your experience installing a backup generator. Is it straightforward or does it require substantial electrical panel rework, especially in older houses?


Annette
Registered user
College Terrace
on Sep 9, 2022 at 1:39 pm
Annette, College Terrace
Registered user
on Sep 9, 2022 at 1:39 pm

Question for the City: if the outages are "rolling" does (or can) this mean that Midtown, Old PA and the Industrial Park are off the hook until all other Palo Alto areas have their turn going w/o electricity? I think it would be helpful to know when my neighborhood or office area is in the crosshairs so that preparations can be made "just in case" an outage becomes necessary. I also think the burden/inconvenience of losing power due to a deliberate shut down should be shared equally by all Palo Alto neighborhoods.


Don't miss out on the discussion!
Sign up to be notified of new comments on this topic.

Post a comment

In order to encourage respectful and thoughtful discussion, commenting on stories is available to those who are registered users. If you are already a registered user and the commenting form is not below, you need to log in. If you are not registered, you can do so here.

Please make sure your comments are truthful, on-topic and do not disrespect another poster. Don't be snarky or belittling. All postings are subject to our TERMS OF USE, and may be deleted if deemed inappropriate by our staff.

See our announcement about requiring registration for commenting.