Getting your Trinity Audio player ready...

Transformers at the East Meadow substation at the intersection of East Meadow Drive and Middlefield Road in Palo Alto on Jan. 31, 2022. Photo by Magali Gauther.

Palo Alto’s effort to light up every nook and cranny of the city with high-speed Internet is slated to roll out this year, with the city preparing to spend more than $30 million to dramatically expand and upgrade the fiber network.

The fiber deployment, which the Utilities Advisory Commission will discuss on Aug. 2, is set to unfold in phases, with the first phase targeting 7,160 utilities customers and 875 businesses, according to a map released by the Utilities Department in advance of the discussion. Encompassing roughly 20% of the city, this phase will cost about $20 million to implement.

But before the city can fulfill the promise of bringing “fiber to the premises,” the Utilities Department has to overcome numerous hurdles that staff say could significantly delay the project. These include negotiating with another Internet provider, AT&T, over use of utility poles that Palo Alto and AT&T co-own under an agreement that dates back to 1918, according to a report from David Yuan, strategic planning manager at the Utilities Department.

Yuan noted in his report that with the city preparing to expand the fiber network and implement full-scale upgrades to its electric grid over the coming years, the agreement may not be feasible to accommodate the increased volume of pole transactions.

Citing the potential for “significant delays” that the AT&T negotiations may spell for the fiber project, Yuan is recommending that the city consider joining the North California Joint Pole Association (NCJPA), a nonprofit that requires its members to abide by cost-sharing methodology for poles.

“In the past, CPAU (City of Palo Alto Utilities) operated without membership with the NCJPA because the volume of pole replacements was manageable by staff and there were additional costs associated with becoming a member,” Yuan wrote. “Given the anticipated increase in pole work, NCJPA membership may be necessary to ensure pole work related to the long-term fiber and electrification projects is conducted in a methodical, timely and cost-efficient manner.”

The AT&T negotiations aren’t the only factor that could slow down the project. The city still needs to conduct an environmental analysis for the fiber expansion, including an evaluation of how the project would impact air and water and whether these impacts could be mitigated.

And it needs to find ways to align the fiber expansion with another massive project: a grid-modernization effort that includes in its first phase upgrading more than 100 miles of the existing overhead electric system as well as various substation transformers. The goal is to “help minimize utility engineering pole make-ready work, pole replacements, noise disruption, and construction activity in neighborhoods” over the next 18 to 24 months, Yuan wrote.

“Given the hundreds of miles of overhead and underground construction activity between FTTP and electric grid modernization, staff is exploring various construction phasing options to minimize disruption, construction activity, and construction costs within neighborhoods while avoiding prolonged deployments,” Yuan wrote.

The fiber project has two major components. One is the construction of a new $26-million “fiber backbone” that would primarily be used for municipal purposes like smart meters and enhanced monitoring of the city’s electric, gas and water services.

The other is a $20-million extension of the fiber ring to local homes and businesses, as long envisioned by local “fiber to the premises” advocates. If things go as planned, the fiber network will turn a profit that would be used to launch future phases in other parts of Palo Alto, ultimately covering most (if not all) of the city.

The city’s fiber plan, according to Yuan, will include installation of about 83 miles of fiber optic cables, which include 48 miles of underground installation and 35 miles of overhead cables that would require use of 6,000 existing utility poles.

The construction of a new fiber backbone and the first phase of “fiber to the premise” will require passing by 1,750 existing poles, attaching cables to 1,300 poles, replacing 100 others and coordinating with third parties to move their equipment lower on another 325.

The city is also beefing up staffing to accommodate the implementation and, ultimately, operation of the expanded fiber system. The current budget adds four positions in Utilities Department dedicated to the effort: an assistant director for customer support services, a manager of information technology and two telecommunications managers.

Even though the first phase of the fiber project falls well short of the city’s ubiquity ideal, it represents a dramatic expansion for the city’s fiber backbone, a small but profitable network that the city installed in the late 1990s and that today consists of 56.3 miles of fiber cable, with 34.6 miles underground 21.6 miles overhead, according to the capital budget that the City Council approved last month.

The neighborhoods that would be included in the first phase of the system’s expansion were chosen based on factors that include lower cost, less competition and higher interest from customers, as gauged from survey responses and the $60 deposits that fiber supporters contributed to the cause well before the project was approved.

In north Palo Alto, the areas chosen for the first phase include a portion of the Downtown North neighborhood just north of Lytton Avenue, between Bryant and Webster Streets; a western section of Old Palo Alto, roughly bounded by Alma Street, Bryant Street, Churchill Avenue and California Avenues, parts of Evergreen Park and College Terrace just south of Stanford Avenue; and the Mayfield neighborhood just east of El Camino Real and north of Oregon Expressway.

In south Palo Alto, the expansion would include a portion of the Barron Park neighborhood east of Bol Park; a small area in the western portion of Midtown and portions of Loma Verde, Charleston Meadows, Palo Alto Orchards and St. Claire Gardens neighborhoods.

Most of the buildout in the early phase would occur in neighborhoods surrounding the city’s two major north-south arteries, El Camino Real and Alma Street. With the exception of a few blocks in Downtown North, the eastern section of Palo Alto is wholly excluded from the first buildout, with no hookups planned anywhere east of Waverley Street, according to a map that is included in Yuan’s presentation.

This map shows in red the sections of Palo Alto that would be prioritized in the first phase of the fiber expansion. Courtesy city of Palo Alto.

The upcoming expansion will represent Palo Alto’s biggest investment in its fiber utility since its debut about 25 years ago. The fiber optics fund, which pays for system improvements, had budgeted expenditures of $1.8 million in fiscal year 2023; $500,000 in budgeted expenditures in fiscal year 2022; and $1.5 million in 2021, according to budget documents. The budget for fiscal year 2024, which began on July 1, includes $33.1 million in expenditures.

In addition to the $20-million fiber expansion — which is being partially funded by the electric utility — the newly adopted capital budget also includes $12.6 million to rebuild the fiber optics network system. The rebuild includes installing substructures and additional fiber backbone cables to increase capacity of section of the network that are currently at or near capacity to meet customer requests for service connections, according to the budget.

The main project areas are Stanford Research Park, the Palo Alto Internet Exchange at 529 Bryant St. and the downtown business district.

Despite the high costs and the prospect of fierce competition from incumbent telecom giants, the council agreed in December to advance the fiber project after more than two decades of debate and numerous false starts. Lydia Kou, who was serving as vice mayor at the time and is now mayor, said she wants the city’s municipal system to provide a “competitive edge.”

“In a way, I think of incumbents as monopolies, and we can provide something similar, or even better,” Kou said shortly before the vote. “I want to make sure everyone in this community does have connectivity. It’s an essential service that people do need.”

Gennady Sheyner covers local and regional politics, housing, transportation and other topics for the Palo Alto Weekly, Palo Alto Online and their sister publications. He has won awards for his coverage...

Join the Conversation

37 Comments

  1. Given that the map is out of date, much of midtown has AT&T fiber. And that AT&T is installing fiber at a fast rate, how can the city justify this?

    Are they going to spend all this money and find out that by they time they are operational, AT&T has beat them to the homes? And they what will they say?

    /marc

  2. Why more faster better expensiver internet? The only advantage to high speed is for gaming. Are we seriously catering to D&D nerds?

  3. How’s the 50 year undergrounding of utilities plan going? Which area is being done at present? Which area is next to be done?

  4. I have to have 2 services because both are so bad, and nothing else is available. It’s not affordable. I wish this was coming to my neighborhood.

  5. “How’s the 50 year undergrounding of utilities plan going? Which area is being done at present? Which area is next to be done? “

    A related question is how will fiber work in neighborhoods already undergrounded? Has CPAU finally answered this question because they hadn’t when this pricey project was pushed through where taxpayers had NO way to reject the project in the city “survey” that immediately requested advanced deposits for a project yet to be approved.

  6. City Fiber has been a big success serving primarily businesses for about 3 decades. That success has piled up $35 million in the Palo Alto Fiber Fund Reserve which is being reinvested now to beef up the fiber network yet again, this time adding selected residental neighborhoods to the offering.
    The big deal is competition.
    Notice how Comcast has gradually improved their service offerings and speed once AT&T Fiber started to be offered in selected Palo ALto neighborhoods.
    The race is on for better fiber services at lower prices in Palo Alto. Couldn’t come soon enough.

  7. I live in one of the neighborhoods that has underground utility lines. Unfortunately, neither AT&T nor the City plan to provide fiber service to us because of the underground lines. Very disappointed. I hope at some point — and sooner rather than later — “every nook and cranny” of Palo Alto, including these neighborhoods, will be served by fiber. It is so much better in terms of speed and reliability, and not just for nerds.

  8. I had a good laugh in reading this article as this has been an ongoing discussion for two decades and some smart people I know have given up trying to advise the city on fiber being deployed throughout our infrastructure.

    I suggest this should be a joint effort with the counties for a mass deployment. And the city is partnering with AT&T for the deployment? Another good laugh although it would be the same with Comcast. Perhaps our elected officials should visit other locations where FiOS has been available for two decades. Or another world renown city such as Chattanooga TN which deployed fiber in their infrastructure years ago.

    Maybe they have better elected officials.

  9. Chattanooga got $112 million subsidy from the federal government and is barely turning a profit. Their profit and economic benefit is almost all coming from their commercial service not residential. Numerous municipalities have lost lots of money on residential fiber that has they need to pay back put of the general fund.

  10. I can’t wait to get the city’s fiber service. Comcast is terrible and I don’t have another reasonably priced option. Go Palo Alto! Let’s get this project moving.

  11. I’m an IT expert with decades of experience. One of my drinking buddies many years ago was Vint Cert before there was an Internet. Vint is well knows as one of the fathers of the Internet, and the Arpanet before it. We are still in touch. When I learned of the stall between Palo Alto with their high-priced Internet ring, and other suppliers like Google and AT&T, I met with a city official to discuss possibilities. The notes of that conversation were published, but seriously taken out of context. I won’t reveal his name, but you may find it online. I gave up on Palo Alto administration, imagining that they have too many private agendas to do the best for we citizens. Already, I risk having my comments deleted!

    My son has AT&T fiber at his location in Palo Alto. It costs a little more than the ancient DSL service, but it is orders of magnitude faster. The big thing most users don’t realize is the service can be as fast uploading as well as downloading, so business users at home can get much better performance with AT&T fiber. My son’s service has been good but not excellent because squirrels sharpen their teeth on the waterproof cable housing. That’s the price we should happily pay for a beautiful city with domestic wildlife.

    I was told that Google offered to install fiber throughout the city a few years ago, and the deal was almost signed. At the last minute, Palo Alto insisted that the entire infrastructure must revert to city ownership and management in a few decades. I was not a party to those discussions, so I can’t tell you if it is truth or fiction. I do know that I want my Internet access managed and supported by a highly qualified organization that will only improve service and reliability. I don’t trust Palo Alto to do that for my lifetime and yours.

  12. I had an unexpected knock on my door this afternoon, and I met two AT&T representatives who signed me up for 2Gb symmetric Internet service over fiber. Unexpected and awesome! I have been waiting for this for a very long time, because I operate a professional remote computer service from my home office. I am now well serviced, and I will make no further comments about how Palo Alto might provide any better service. This is one of those rare times when I think “good enough” wins.

  13. @Homer Jamison, fascinating tale. And given the way the city’s been aggressively pushing for city ownership of a competitive business requiring real-time customer service — something at which they’ve never excelled –, they must be convinced there’s some benefit to staff, consultants etc — in managing all the outsourcing contracts.

    The question is what.

    Another Vint Cert buddy had tales to tell about his early involvement with the city’s earlier internet connectivity boondoggledecades ago where he used to joke that the whole process from start down to the flawed installation manuals required each customer to have their very own dedicated high-level engineer.

  14. More $tudie$ Needed. Measurements of the orifice of the pocket, and calculations of how many Benjamins it will it take to fill the hole. Every time this subject hits the front page, the orifice is bigger than last time.

  15. I would happily pay an extra installation cost to get fiber to my home, but AT&T doesn’t seem willing to deal with parts of Palo Alto with underground utilities.

    It seems like this is an area where the City of Palo Alto could get a lot of new fiber customers, as there is a lot of pent up demand (I know of 3 neighbors who have been waiting for fiber for years), and in this part of Palo Alto, the city will have no competition from AT&T for fiber service.

  16. @JohnMcD, it would have been if CPAU Fiber oeople were even aware that some neighborhoods had underground wiring. When I wrote to ask them about it early on, they asked ME which neighborhoods those were!!!

    Also, if Palo Alto will be piggybacking on AT&T, awhy would AT&T suddenly become more “willing to willing to deal with parts of Palo Alto with underground utilities’ under its outsourcing contract with Palo Alto.

    As someone who paid for undergrounding decades ago, I resent having to pay for a service I can’t even use — especially one so poorly thought out as this one.

  17. @Online Name
    Re: “As someone who paid for undergrounding decades ago,…”

    Don’t forget that the formerly undergrounded neighborhoods will be de-undergrounded as the city updates/expands the local electrical supply network. Large green metal electrical boxes will pop up like 2001’s monoliths in your front yard, emitting a pleasing buzzing sound.

  18. Sonic.net is advertising 10 Gb/sec fiber for $50/month. The only problem is that they don’t service Palo Alto as of now. But I believe they are interested. I have been a Sonic customer for many years and now with fiber. When I got fiber it was through AT&T. They came and did the installation. As of the first of the year Sonic is not affiliated with AT&T and so I’m grand-fathered in. The thing I like about Sonic is their customer service. If you have a problem, you talk to someone now, no pushing buttons to switch you around. I also have VOIP for my land line. Are the city fiber people talking to Sonic.net for Palo Alto’s needs?

  19. Actually Sonic does provide access to Palo Alto; we’re current subscribers to Sonic which piggybacks on the AT&T lines, the same ones Palo Alto plans to use.

    Unfortunately AT&T has been down since around 8:30 AM with no restoration time given and Sonic says to call back tomorrow if AT&T hasn’t fixed the problem by then.

    We’re usually pleased with Sonic’s customer service.

    (I’m connecting now via my cell connection.)

  20. AT&T did me a solid today! I now have 2 gigabit Internet service at my apartment in the Ventura neighborhood, and it tests out at about 2.5Gb in both directions. With five static IP addresses, I expect my monthly bill to be around $125.

    Why 2Gb you may wonder. It gives me a way to test and tune the higher end performance of the new WiFi 6E access points. Some kinds of progress come slowly to the heart of Silicon Valley, but they’re really coming.

  21. Thanks. I told my partner and he says we’re probably on different fiber nodes and that someone from our neighborhood who’s also experiencing an outage thinks construction on Seale may have cut a cable line. (Sigh)

  22. An AT&T fiber line runs past our house, underground along Middlefield Road. Our internet service went down at about 1:30 AM on Saturday, July 29, and it’s still not working as of Sunday morning.

  23. This 2-day outage in Leland Manor is being widely discussed on NextDoor.

    No doubt our $30,000,000 will ensure that AT&T’s customer service for Palo Alto fiber customers is so much better than what they currently provide, right Palo ALto??

  24. Got ATT fiber a while back. Absolutely wonderful! We’ve never had an outage and the price is good. Do you really want local govt involved in this??? Remember what Reagan said: run the other way if they say they are here for the government to fix something.

  25. Just fyi that much of Leland Manor is STILL down on Monday evening and has been down since Saturday.

    None of the folks discussing this on Nextdoor got any type of email or answers from AT&T any warning from AT&T nor have we gotten any estimates on when service will be restored.

    Wondering how PA will be staffing up for customer service and AT&T account management.

  26. Leland Manor is STILL without AT&T internet access and without ANY response from AT&T, including when we might see service restored.

    People who’ve signed up for text alerts have heard nothing.

    People are wondering if the trenching/construction on Seale a few days ago caused the outage.

    Is this the type of customer DISservice we can expect for $30,000,000 or will the city use this as negotiating leverage.

  27. > Re: Online Name: “if Palo Alto will be piggybacking on AT&T, why would AT&T suddenly become more ‘willing to willing to deal with parts of Palo Alto with underground utilities’ under its outsourcing contract with Palo Alto.”

    I’m not saying AT&T would start caring about my neighborhood— they clearly don’t care about it now! My neighbors and I have been asking for fiber service for years from them and they just don’t seem to care about areas with underground utilities.

    My hope is that CPAU Utilities doesn’t just compete directly with AT&T. There will be more demand and higher willingness to pay in neighborhoods like mine that AT&T Fiber doesn’t already serve and thus have no current option for fiber.

  28. @JohnMcD, it would be soecial if AT&T cared about its direct customers and those of its partners/distributors like our Sonic.net all of whom have been struggling for days to get any type of answer from them.

    One persistent NextDoor poster reported today “ I just reposted this with an update from my call with ATT today; they are now reporting an estimated resolution on AUGUST 6 – five more days! ???? And who knows if they make it. I’m now at Rinconada Library taking advantage of their excellent Wi-Fi and DVD collection.”

    Today Sonic.net finally got a non-answer from AT&T that it could tell Somic nothing about when service would be restored — four days into the outage.

    One would hope the city and our local media are on top of this if they hope to pitch PA Fiber to The Home as a reliable service for people and businesses.

    (Let’s hope this post goes through and that cell phone is considered a secure connection.)

  29. Thanks to the perseverance of several neighbors on NextDoor our service has been restored after only 5 — FIVE — days.

    AT&T’s customer service was a bad joke, especially the agent who told us service wouldn’t be restored until Sunday.

    Be prepared for a rocky, financially risky and costly ride.

Leave a comment