Why Is San Francisco So Expensive?

I plan to write a lot about San Francisco's housing woes and stratospheric cost of living, but I thought I would kick off with a tale of two photos.

This is the Castro in San Francisco, one of that city's most highly sought after neighborhoods:

This is 8th Avenue in Manhattan:

San Francisco's current predicament is highly complex. But, if you want a simple answer to why it's so expensive, that answer is: lack of density.