Yet another “Questionable” Staff Report/Survey

At Monday’s Council meeting (12/9), the City Manager will present an Informational Report on a Transportation Survey (Web link) performed in Spring 2013. This is the sort of report that would be get you laughed out of the room in many private companies, and it is amazing that it wasn’t pulled back after the fiasco of the previous Council meeting (12/2): “Traffic numbers prompt council head-scratching: Planning department graph shows less traffic now than a decade ago” (Web link).

The first, and very basic problem is that the respondents were self-selected, but Staff makes no apparent attempt to determine how representative they were, nor does the report alert the reader to this potential problem. Second, the questions in the survey did not seem to be adequately designed. I suspect that the results and the interpretation may be heavily skewed.

Because performing a survey with a statistically valid sample is typically expensive, often you can only afford one with self-selected respondents, and such surveys are often better than nothing. However, that doesn’t excuse not trying to estimate how representative they are.

Palo Alto has a population of approximately 65,000 people and 26-29,000 households. There were 2855 responses from residents. Since the questions were geared to households, if each of these responses represented a separate household, that would be an 11% response rate, which is acceptable for such surveys. From the graph presented for question 6 on commutes, it appears that roughly 30 respondents said that they didn’t work, that is, roughly 1% of the respondents. Ding, ding, ding. A large part of the Palo Alto population is not represented by this survey.

A major part of this survey was for people commuting to work within Palo Alto, and that response rate was less than 2%.(f1) This seems grossly inadequate, and definitely should have been prominently noted.

This type of survey needs to take into account skews induced by how it was publicized. One important means was the neighborhood email lists and related social media. Having used these means to promote various surveys of my own, I have observed that there are a few neighborhoods where you get very good response rates and many neighborhoods where you get few or no responses. The City’s survey did not honor these natural groupings, making it impossible to see such effects. My neighborhood–Barron Park–is one that produces excellent response rates, but the City split it into two zones, grouping each with neighborhoods that have had very poor response rates. This may well hide that substantial portions of the City are very poorly represented in the results.

Even for the geographic zones the City chose to use, the reporting comes up short: It reports number of (self-identified) responses for each zone without normalizing for the population of those zones (either people or households).(f2) Similarly, the Employee Survey made no attempt to normalize the results. Alarm bells should have rung when the responses from Stanford Research Park were only 40% of those from the University Avenue Downtown and only 50% more than those from California Avenue.

Yet another sign that the responses were heavily skewed or otherwise biased was the report that 20% of residential vehicles use alternative fuels. My guess would have been far, far less. As a sanity check, I walked in my neighborhood late Saturday afternoon and counted 84 parked cars (driveways and on-street) and spotted only 2 hybrids and 1 electric (Tesla). 22 non-commercial vehicles passed me, and all were using gasoline engines. I fully expected my count to be less than the true average, but the disparity is much too large. Furthermore, is it credible that 11% of private cars in Palo Alto use CNG (Compressed Natural Gas)? CNG is predominantly found in commercial fleets. Given the popularity of hybrid cars, I find it wildly implausible that there are only 50% more hybrids than CNG private vehicles. How could this not set off alarm bells?

Various of the questions seemed more driven by ideology than desire to inform policy. For example, the question about number of bikes you own has no relationship to your using bikes around town. I know people whose bikes are solely for recreational use (touring, wilderness), and others who have given up riding, because of safety concerns, or because the City’s development policies have pushed their destinations too far away to be bike-able, or…

For example, there is a question about whether you live within a mile of a Caltrain station.
1. The Staff report forgets the San Antonio station (because it isn’t quite in Palo Alto?).
2. It assumes that people reported accurately–I had to use GoogleMaps to determine my distance, but I suspect few respondents were that curious or concerned about accuracy.
3. According to Staff, I live just within that 1-mile distance, but I view myself as living 1.4 miles away. Crows may fly a straight line, but as a person, I have to follow the street grid.
4. In discussions of why people don’t use Caltrain, raw distance is far down the list. Safety along the route to&from, is a much larger issue, especially for women.
Yet my experience has been that it is very hard to get Staff to pay attention to the practical impediments–the ones that affect real people–instead of focusing on bureaucratically satisfying actions.

With the prominence of the problem of employees parking in residential neighborhoods, I expected the survey results for the question on where employees park (pg 23 of the report) to break out the results by business area. Instead we are shown a chart where the on-site parking in the Research Park and similar locations swamps the results.

But wait. I forgot that Staff reports aren’t intended to provide factual bases for decision-making, but merely to provide a rationalization of what Staff thinks should be done.
Nevermind.

—- Footnotes —-

1. Employee response rate: There were 852 non-resident respondents plus 39% of the residents reported working in Palo Alto or on the Stanford campus. The estimates of employees in Palo Alto vary significantly, but approximately 100,000 is relatively non-controversial.

2. Normalizing: The City has a Geographic Information System (GIS) that should be able to produce a very good approximation of these numbers.

—-
The Guidelines (Web link) for comments on this blog are different from those on Town Square Forums. I am attempting to foster more civility and substantive comments by deleting violations of the guidelines.