Ok, here are my results so far, maybe it will also help someone in future with similar problems:
The Bios Update of the ASUS P5VDC-MX was applied. It did not help with the CRTEmudriver 2.0 Setup. So I removed all ATI drivers again and pointed windows manually to the driver directory. It won't install and tell me VGA default adapter has the most recent driver. So I did the same with CRTEmudriver 1.2 and with this driver the video card is detected as an AGP Version, the driver installs, so I can use it now with Testsigning enabled. I use VMMaker 1.4 and have the desktop displaying fine on my CRT with the old 1.2 driver. The newest Groovymame starts and runs fine, but I have to see if I can make it switch resolutions, because for now it just seems to use the desktop resolution. If it does not, I think I am still fine and will just install my most used resolution and select them manually - so I am happy anyway.
The 2.0 setup still cannot find a compatible device, but I am not entirely sure if its really a bug with the detection or a problem with my somewhat broken combo of card and mobo.