I couldn't find a captcha in a popup window on the live url you provided.
The desired scenario can be achieved using the RadWindow control, as a popup window containing the RadCaptcha control. For your convenience I have implemented a simple project showing this behavior. Please find it attached to the thread.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items