Use the flat end of a spudger to lift the joystick up and out of its recess.

Use your fingers to remove the joystick.

There is a thin black gasket around the hole where the joystick pokes through the frame. Try not to disturb this gasket as you remove the joystick.