Yet another "Questionable" Staff Report/Survey | A Pragmatist's Take | Douglas Moran | Palo Alto Online |

Local Blogs

A Pragmatist's Take

By Douglas Moran

E-mail Douglas Moran

About this blog: Real power doesn't reside with those who make the final decision, but with those who decide what qualifies as the viable choices. I stumbled across this insight as a teenager (in the 1960s). As a grad student, I belonged to an org...  (More)

View all posts from Douglas Moran

Yet another "Questionable" Staff Report/Survey

Uploaded: Dec 7, 2013
At Monday's Council meeting (12/9), the City Manager will present an Informational Report on a Transportation Survey (Web link) performed in Spring 2013. This is the sort of report that would be get you laughed out of the room in many private companies, and it is amazing that it wasn't pulled back after the fiasco of the previous Council meeting (12/2): "Traffic numbers prompt council head-scratching: Planning department graph shows less traffic now than a decade ago" (Web link).

The first, and very basic problem is that the respondents were self-selected, but Staff makes no apparent attempt to determine how representative they were, nor does the report alert the reader to this potential problem. Second, the questions in the survey did not seem to be adequately designed. I suspect that the results and the interpretation may be heavily skewed.

Because performing a survey with a statistically valid sample is typically expensive, often you can only afford one with self-selected respondents, and such surveys are often better than nothing. However, that doesn't excuse not trying to estimate how representative they are.

Palo Alto has a population of approximately 65,000 people and 26-29,000 households. There were 2855 responses from residents. Since the questions were geared to households, if each of these responses represented a separate household, that would be an 11% response rate, which is acceptable for such surveys. From the graph presented for question 6 on commutes, it appears that roughly 30 respondents said that they didn't work, that is, roughly 1% of the respondents. Ding, ding, ding. A large part of the Palo Alto population is not represented by this survey.

A major part of this survey was for people commuting to work within Palo Alto, and that response rate was less than 2%.(f1) This seems grossly inadequate, and definitely should have been prominently noted.

This type of survey needs to take into account skews induced by how it was publicized. One important means was the neighborhood email lists and related social media. Having used these means to promote various surveys of my own, I have observed that there are a few neighborhoods where you get very good response rates and many neighborhoods where you get few or no responses. The City's survey did not honor these natural groupings, making it impossible to see such effects. My neighborhood--Barron Park--is one that produces excellent response rates, but the City split it into two zones, grouping each with neighborhoods that have had very poor response rates. This may well hide that substantial portions of the City are very poorly represented in the results.

Even for the geographic zones the City chose to use, the reporting comes up short: It reports number of (self-identified) responses for each zone without normalizing for the population of those zones (either people or households).(f2) Similarly, the Employee Survey made no attempt to normalize the results. Alarm bells should have rung when the responses from Stanford Research Park were only 40% of those from the University Avenue Downtown and only 50% more than those from California Avenue.

Yet another sign that the responses were heavily skewed or otherwise biased was the report that 20% of residential vehicles use alternative fuels. My guess would have been far, far less. As a sanity check, I walked in my neighborhood late Saturday afternoon and counted 84 parked cars (driveways and on-street) and spotted only 2 hybrids and 1 electric (Tesla). 22 non-commercial vehicles passed me, and all were using gasoline engines. I fully expected my count to be less than the true average, but the disparity is much too large. Furthermore, is it credible that 11% of private cars in Palo Alto use CNG (Compressed Natural Gas)? CNG is predominantly found in commercial fleets. Given the popularity of hybrid cars, I find it wildly implausible that there are only 50% more hybrids than CNG private vehicles. How could this not set off alarm bells?

Various of the questions seemed more driven by ideology than desire to inform policy. For example, the question about number of bikes you own has no relationship to your using bikes around town. I know people whose bikes are solely for recreational use (touring, wilderness), and others who have given up riding, because of safety concerns, or because the City's development policies have pushed their destinations too far away to be bike-able, or...

For example, there is a question about whether you live within a mile of a Caltrain station.
1. The Staff report forgets the San Antonio station (because it isn't quite in Palo Alto?).
2. It assumes that people reported accurately--I had to use GoogleMaps to determine my distance, but I suspect few respondents were that curious or concerned about accuracy.
3. According to Staff, I live just within that 1-mile distance, but I view myself as living 1.4 miles away. Crows may fly a straight line, but as a person, I have to follow the street grid.
4. In discussions of why people don't use Caltrain, raw distance is far down the list. Safety along the route to&from, is a much larger issue, especially for women.
Yet my experience has been that it is very hard to get Staff to pay attention to the practical impediments--the ones that affect real people--instead of focusing on bureaucratically satisfying actions.

With the prominence of the problem of employees parking in residential neighborhoods, I expected the survey results for the question on where employees park (pg 23 of the report) to break out the results by business area. Instead we are shown a chart where the on-site parking in the Research Park and similar locations swamps the results.

But wait. I forgot that Staff reports aren't intended to provide factual bases for decision-making, but merely to provide a rationalization of what Staff thinks should be done.

---- Footnotes ----

1. Employee response rate: There were 852 non-resident respondents plus 39% of the residents reported working in Palo Alto or on the Stanford campus. The estimates of employees in Palo Alto vary significantly, but approximately 100,000 is relatively non-controversial.

2. Normalizing: The City has a Geographic Information System (GIS) that should be able to produce a very good approximation of these numbers.

The Guidelines (Web link) for comments on this blog are different from those on Town Square Forums. I am attempting to foster more civility and substantive comments by deleting violations of the guidelines.
What is it worth to you?


Posted by GIGO, a resident of Downtown North,
on Dec 8, 2013 at 5:04 pm

Garbage In Garbage Out of numerical models is called GIGO. City analysis is biased, manipulated and so flawed that their models spit out garbage. They report the results staff wants to justify their decisions. Reality is completely ignored. It's not just this week's traffic study, but all the Staff Reports produced over the last few years. Either staff is incompetent or corrupt. No other rational explanation exists.

In the absence of good data and trustworthy analysis there is no possible way for Council or the public to make sound decisions.

Posted by PA TBOT, a resident of Green Acres,
on Dec 9, 2013 at 2:42 am

Just before the election, Gail Price insisted only a minority in the neighborhood were against the rezoning (ultimately 80% of the votes cast in the closest neighborhood to the property were against). I saw posts about that time claiming a survey showed Palo Altans were two to one For.

Someone did an intrusive survey by a professional organization when the referendum was qualified. Word quickly got around about it and anyone against wouldn't answer unless the caller identified who was sponsoring the survey. It was an attempt to leverage the info, but it never worked, and thus, very few people on the against side were ever interviewed.

On the other hand, internal informal polls in the same neighborhood pretty well aligned with such overwhelming opposition.

Trustworthy polls start with trustworthy motives. The neighbors really just wanted to figure out how people felt.

Posted by Margaret Fruth, a resident of Ventura,
on Dec 11, 2013 at 10:49 pm

Margaret Fruth is a registered user.

Commute time is also very important, possibly second only to safety.

Comparisons of straight line commutes versus street grid omitted data for rocket cars & this month only, the occasional sleigh with flying reindeer.

Follow this blogger.
Sign up to be notified of new posts by this blogger.



Post a comment

Sorry, but further commenting on this topic has been closed.

Stay informed.

Get the day's top headlines from Palo Alto Online sent to your inbox in the Express newsletter.

Marriage Interview #17: They Renew Their Vows Every 5 Years
By Chandrama Anderson | 12 comments | 2,171 views

Tree Walk: Edible Urban Forest - July 8
By Laura Stec | 5 comments | 1,410 views

New Zealand-inspired savory pie bakery coming to Montara
By The Peninsula Foodist | 2 comments | 1,262 views