make sure the RAM you are using meets the specs listed in this Apple Knowledge Base article:
http://support.apple.com/kb/HT2266
PowerPC G5 processors are very finicky about RAM, make sure you are using quality RAM with a lifetime warranty - don't buy the cheapest module out there. Sounds like your machine just doesn't like the 2GB module... hopefully you can return it? If so try a different brand: OWC, Crucial, EDGE, Kingston all offer RAM which works well with G5 processors.
You cannot remove the internal 512MB module - it is surface mount soldered onto the logic board.