The problem is not damaging the pad, it's removing the plating on the edges of the hole. On a double sided board with plated-through holes, the edge of the hole is plated, forming a connection between the pad on the top and the pad on the bottom. If you damage this plating, the connection will be broken.
/mike