Add the M5 bolt with a washer on it,

Make sure the washer on the bearing is still there, it's very important. You can use a flat screw driver to help you push it at the right place.

The internal and external washer dosent need to be precise 1mm. Only the one in the "middle".