Generally we do not guarantee that our controls will work in combination with other third party controls. We have not tested the FancyBox, so not sure what the problem might be.
You can inspect the page after clicking the button, to see what styles are applied to the view and the box and to see if the Z-index of the view is not smaller than this of the box. In this case it will be shown under the box and will be invisible.
See this help topic about the Z-indexes of our controls:
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed