It tends to be that way every time, in spite of all the talk about "legacy" used to promote the Olympics. I went back to the Nordic venue two years after the '92 Olympics, and the place looked like something after a war had wiped humanity out: broken windows in the press building, doors hanging off the hinges...
Someday they need to reconsider the idea of a permanent international venue(s) with cost shared among participating countries. This might also take a little of the overt nationalism out of the Olympics and focus it back on that forgotten but, let's face it - useful group: the athletes.