That's not completely true. Most of the major cities had at least their centers severely damaged, some much more heavily than others (compare, for example, Leipzig, that was only hit piecemeal, with Dresden, whose Altstadt was obliterated). Some cities, like Bonn, Weisbaden and Heidelberg escaped almost completely untouched. Many smaller towns and villages were left alone and bypassed. Very few unlucky cities were completely destroyed
So although some of the old buildings needed to be rebuilt from almost nothing (most of Dresden's Altstadt again, Munich's old Rathaus, Berlin's recently completed Stadtschloss), a lot of the preservation of old buildings you see today was performed to limit age-related decay, not necessarily rebuilding from war damage. Some of the old buildings actually look better today than they did before the war, mainly because historical preservation is more valued now than in the past. This process has really picked up since reunification, especially in the old east, where many of the older structures were left to rot, even without war damage.