Steven,
I'm in the same boat as you, looking for a first plane (IFR 172).
I think the real concern here is resale. If you plan to keep the plane a long time and you're personally satisfied of its airworthiness, then there's no issue. However, there are a certain number of buyers who will categorically rule out any plane with damage history. That's going to reduce your pool of buyers somewhat.