You need to turn off hardware acceleration to perform a screen-recording of a DVD playback. Apparently, WinDVD does not work when hardware acceleration is switched off.
We have seen that Cyberlink's PowerDVD works even when hardware acceleration is turned off. Also, My Screen Recorder is able to record DVD played back by PowerDVD.