The real reason for the prohibiting is not that a wide shoulder on a 45 mph arterial road is "unsafe" for bicyclists and walkers, as claimed by County highway engineers. The real reason for prohibiting is political, that County highway engineers want to keep their options open to turn expressways into future freeways, as the 4-lane Guadalupe Expressway/Parkway in San Jose was changed into a 6-lane freeway (Freeway 87), kicking the bicyclists off in the process. Even for expressways that are unlikely to be future freeways, they oppose use of ALL shoulders, no matter how wide, for consistency reasons, in order not lose their false "shoulders are unsafe for bicycles and pedestrians" argument. If a shoulder would be 35 feet wide, they would still claim "unsafe" for no other reason than they can get three extra car lanes out of that in future lane addition projects. Note: as of 1991, the County highway engineers lost their final fight against bicyclists, and no longer make "unsafe" claims against bicyclists using shoulders and bike lanes, only against walkers and transit patrons.
City ordinances were enacted circa 1965 at the urging of the Santa Clara County highway engineers which attempted to ban pedestrians and bicycles on expressways, despite the fact that, for most expressways, the ordinances contradicted State Law. The stated purpose of enacting prohibitions on what were mostly existing roads was the intention for them to be 65-mph freeways. The funding was never obtained to convert the roads into freeways, nor to acquire the right of access from all abutting property owners that is a requirement for freeways.
Ed Stefani, Principle Engineer for the County in the 1960s, vigorously opposed use of shoulders even by bicyclists, according to old-time members of Almaden Cycle Touring Club. In the 1980s, Ed Stefani stated that instead of freeways, expressways became "glorified streets." Yet, County highway engineering and city traffic engineering departments --excepting only those in Mountain View, Palo Alto and, for walkers only, Sunnyvale-- opposed the repeal of bicycle and pedestrian prohibitions (in their respective cities) despite the fact the roads were no longer planned to be freeways.
A never-stated reason for prohibiting non-motorists was this: When adding lanes in the future, minimize costs to the lane-additions project by not relocating bicycle/pedestrian facilities. At some locations, destruction of existing pedestrian facilities, instead of relocation, did occur on three expressways. This has been a common practice among highway/traffic engineers, according to a new FHWA document which opposes the practice. Here are several cases from the County:
One case is the highway engineers' prohibition of the pedestrian underpass going underneath Central (at Lawrence Expressway) in Santa Clara. This route, with the only alternative being jay-walking across Central, was and is used by pedestrians walking from the Lawrence Caltrain station to nearby industry. Before repeal, half the people jay-walked while the other half ignored the "pedestrians prohibited" signs and used the underpass. The prohibition was repealed by the Santa Clara City Council in 1991, despite four years of vigorous opposition by the highway engineers. When adding lanes circa 1995, it would have been cheaper to just bulldoze the pedestrian underpass instead of rebuilding it. The repeal forced them to reconstruct it, and also forced them to construct sidewalks along Lawrence because the shoulder, legally used by pedestrians, was being eliminated for additional motorist lanes. (The shoulders were subsequently restored for bicycles.)
A demonstration of the highway engineers' unwillingness to use roadway funds for pedestrians was on Montague Expressway in San Jose in 1989. Sidewalks were destroyed when adding lanes instead of relocating or reconstructing the sidewalk. Only years later, after money was allocated from pedestrians funds (the TDA account), was a sidewalk rebuilt, and only on one side. Using TDA funds for lane-addition projects is the same as eliminating an equivalent length of sidewalk from being constructed elsewhere. Use of TDA funds to construct traffic lanes is wrong. It is stealing money from another intended purpose. Pedestrian funds should have gone to projects that benefit pedestrians. Roadway widening only benefits motorists.
More cases are described in the main report under Prohibitory ordinances have affected roadway designs and are very damaging to pedestrians.
The expressways were paid for by general taxation, a County bond measure repaid by property taxes, instead of motorist use fees. Most were existing public roads that had lanes and shoulders added, and a name change to "expressway." Highway engineers need to use roadway funds, not pedestrians funds, on expressways because safe pedestrian facilities, the wide shoulders, were all paid for by general taxation. If they eliminate shoulders to add new lanes, then the lane-addition project's funding needs to be used to build sidewalks or paths. Instead, highway engineers have often resorted to the cheapest solution: posting of "Pedestrians Prohibited" signs.
Allowing pedestrians on shoulders of even one expressway today gives recognition to the fact that shoulders on all expressways are valid pedestrian facilities where alternatives are lacking, the usual case. Allowing use of even one shoulder, no matter how wide, would prove their "shoulders are unsafe" argument to be false. Therefore, no matter how wide a shoulder is, they still claimed that shoulders are "unsafe" while deliberately ignoring the fact that alternative routes are usually much riskier and/or impractical.
Their prior opposition to pedestrian use of pedestrian paths [photo] is harder to rationalize. But consider this fact: Rollo Parsons, Branch Manager, Highway Design, County Roads&Airports Department, refused to remove "pedestrians prohibited" signs on well-used, wide pedestrian paths on Capital Expressway in San Jose in 1996, despite the fact that the County Counsel rendered a legal opinion stating the signs are illegal. It took one year of MTS fighting them, necessitating going to the BPAC and a County Supervisor, to achieve compliance. (Rollo Parsons also stonewalled for one year on removal of "bicycles prohibited" signs in Sunnyvale in 1987. It took one year of fighting by MTS after the repeal by Sunnyvale.) More details about opposition to paths are in Pedestrian path creation; questions for County highway engineers.
Most illogical of all, and very unsafe, is where pedestrians were allowed on shoulders yet they eliminated the shoulders by changing them into traffic lanes, forcing pedestrians to walk in the traffic lane. This occurred on three expressways. (Subsequently, two had sidewalks constructed at the persistence of MTS.)
New (2005): See the County highway engineer's triple attack on walkers and transit patrons
A new FHWA document recognizes that shoulders are "a place for bicyclists and pedestrians to operate."
More answers to the title question are in Conflict of Transportation Competitors by MTS and an excellent general analysis of bicycle/pedestrian prohibitions in the USA by the Victoria Transportation Policy Institute, titled "Whose Roads? Defining Bicyclists' and Pedestrians' Right to Use Public Roadways." [Download the pdf file from their Pedestrian and Bicycling Issues page.]