How to remove joystick caps

I have ordered new joysticks for my lite, but wanted to know if it was possible to remove the original joystick caps?

I have a pokemon edition lite with the non standard joystick colors and they are still in mint condition (minus drift). So was hoping to move them to the new joysticks.

No amount of pressure let the cap come off.
Seeing as the drift had already killed the joystick I decided to tear it down.
It appears that Nintendo has the stick and cap molded as a single unit in production, so its not possible to get the cap off to swap onto the new Guli sticks.

At best I might be able to get just the rubber cover off, but no idea if that would fit the Guli caps.
I am happy to have my sticks working again, but also sad that I am losing the unique coloring.