The five-member panel, which consists of professors and transportation experts, found that the ridership model, while "generally well founded and implemented," suffers from a series of major flaws. These include insufficient consideration of socioeconomic factors; a bias in the survey data used as a basis for the model; and a failure to distinguish between short and long trips when calculating the impact of schedule delays.
The highly technical report, which was released last week and covers the panel's findings and recommendations during its January to March review period, confirms earlier findings from the UC Berkeley Institute of Transportation Studies and from the Palo Alto-based watchdog group Californians Advocating Responsible Rail Design (CARRD). Both groups had criticized the methodology used by the consulting firm Cambridge Systematics and argued that the California High-Speed Rail Authority's estimates of the number of people who would ride the rail system are too flawed to be used for setting policy.
The panel, which reports to rail authority CEO Roelof Van Ark, is chaired by Frank Koppelman, professor emeritus of civil engineering at Northwestern University. It also includes Kay W. Axhausen, a professor at the Institute for Transport Planning and Systems in Zurich, Switzerland; Billy Charlton from the San Francisco County Transportation Authority; Eric Miller, a professor of civil engineering at the University of Toronto; and Kenneth A. Small, a professor emeritus in economics at University of California, Irvine.
The panel calls Cambridge's ridership model "ambitious" and representing a "significant improvement in practice in several respects." But the report also notes that "there are important technical deficiencies in the model and the documentation thereof." It encourages the rail authority to lower its projections.
"Pending improvements to the model, we recommend that any use of the model include some steps to make the demand forecasts more conservative, especially in forecasts for financial (investment and risk) analysis," the report states.
A rail official, meanwhile, said that the report highlighted the complex nature of forecasting.
"Essentially, the report says that this is the most ambitious and most transparent modeling exercise to have occurred in this realm and, when refined, will represent best practices for this kind of forecasting in North America. The panel asserts no bias or improper practices," spokesperson Rachel Wall said in an email.
"What the panel expresses desire for is more documentation and more testing, both of which have been provided to the panel in the time between March and today, and which will be reflected in the forthcoming reports from the panel," she said.
One flaw that the panel identified involved the ridership model's treatment of out-of-vehicle travel time, particularly the time passengers have to wait when trains are delayed. The report states that the assumptions used in the Cambridge model to calculate the "constraint on out-of-vehicle travel time" are valid only for urban trips with small headways (that is, the distance and time between trains). The report cites a study showing the passengers' behavior is much different in the "intercity market." In other words, passengers are much more likely to stomach scheduling delays if they're preparing for long trips out of town as opposed to jaunts from one neighborhood to another.
The report calls the ridership model's use of this constraint "unjustified."
The panel also found "several instances of incomplete or outdated information in the documentation," according to the report. This includes insufficient discussion of such factors as fare levels, highway and airport congestion, train frequency and analysis of how the proposed train system would impact other modes of transportation, including airlines and intercity bus services.
The report is particularly critical of the survey used by Cambridge to get data for the ridership model. The company used a technique called "choice based sampling" which targets and, as a result, over-represents a specific subset of the population (in this case airline and train travelers). The firm conducted surveys in 2005 at airports, rail stations and over the phone. This included on-board surveys on Altamont Commuter Express trains, telephone interviews of Amtrak passengers and surveys of passengers at six California airports.
The report notes that while choice-based sampling is useful for making sure "enough respondents were found to choose each of the major modes," the technique is also "known to bias estimation results unless the estimation procedure is modified to take account of this sampling."
"The method used by CSI, which was believed to be correct at the time of model estimation, has since been shown to be incorrect and a new procedure has been developed which is correct," the report states. "Future estimation work should take advantage of this new knowledge."
The panel released its findings at a time when the rail system continues to weather criticism and financial uncertainty. State senators Joe Simitian and Alan Lowenthal have consistently criticized the rail authority's ridership forecasts and business plans over the past two years. Other critics, including CARRD and a group of Silicon Valley economics and business executives, issued reports criticizing the rail authority's business plan and its assumptions about federal grants and private investments.
In addition, the rail authority communications efforts have taken a hit in the past month, with the resignations of both its PR firm, Ogilvy Public Relations Worldwide, and its deputy executive director in charge of communication, policy and public outreach, Jeffrey Barker. Barker had been appointed by Gov. Arnold Schwarzenegger in 2009.
A representative from CARRD said Saturday that transparency is missing from the state agency.
"You can fire your PR agency and your head of communications can leave, but in the end, someone is telling them what to do. There needs to be a giant culture change to fix the systemic transparency issues, and we don't know who, if anyone, at the authority is truly interested in that goal," Nadia Naik, co-founder of CARRD, stated in an email.
However, Wall, of the rail authority, asserted that the agency is devoted to providing information to the public.
"The California High-Speed Rail Authority is committed to transparency and strives to ensure access to the project details and documents through various platforms including its website, webcasting, public meetings and more," she said.
On Wednesday, CARRD had gone public with its then-unsuccessful attempts to get the authority to release information under the California Public Records Act request. Among the requests was the peer-review panel report.
The initial request was made March 22. Despite a requirement that a decision on the request be made within 10 days, or a reason is provided that the records cannot be released, the authority failed to follow through on the request for a full three months, Naik said.
In the Wednesday letter to the rail-authority's board, CARRD co-founder Elizabeth Alexis detailed the communications between CARRD and the rail authority. In late April, Barker told CARRD that the peer-review committee had not submitted documentation of its January through March deliberations to the authority. Then in mid-May, he said that the requested information would be available at the end of the following week. In early June, he again said that documents would be forthcoming. And in mid-July, he said that there were no documents, only drafts, which could not be released, Alexis wrote. In addition, she said, Barker commented that CARRD just wanted to make the rail authority look bad on CARRD's website.
On Thursday, the day after CARRD's letter to the board and the same day that Barker announced his resignation, the authority released the report.
READ MORE ONLINE
The peer-review panel's report has been posted on Palo Alto Online. To read it, search under "Panel finds flaws in high-speed-rail forecasts."