In Europe, I was living for several years just 500ft from the high speed rail line connecting Paris to Frankfurt. The most I heard was an occasional distant whoosh, far quieter than normal street traffic. High speed rail lines are quieter and less disruptive than the slow, outdated, and noisy Caltrain trains. The berms on which those tracks run also were no more of an eyesore and no more disruptive than Caltrain's current tracks in Palo Alto. And they would reduce the danger and disruption from the existing railroad crossings. I don't think tunnels are either necessary or cost-effective; even in cities like Berlin and Frankfurt, long distance trains usually run above ground even in city centers.
Obstructing sensible and standard construction of high speed rail lines through Palo Alto has numerous risks. In the worst case scenario, the delays may threaten federal funds and/or the line might simply bypass Palo Alto and the peninsula altogether. Or, the line may go through Palo Alto, but we get no stop on the high speed rail system.
Overall, in terms of impact on the city and neighborhoods, I think high speed rail on an above ground berm would be neutral or even an improvement over the current Caltrain system for Palo Alto. Palo Alto should focus on ensuring that there will be a high speed rail stop in the city, on maximizing the economic benefits to the city, and on obtaining compensation for those residents whose lives will be disrupted by construction.