You can try replacing Telerik.Web.UI.DialogHandler.aspx
in your web.config file.
Also, if you have spellcheck enabled, you need to register its HTTP handler as shown in this help article
I hope this helps. Feel free to contact us again if you run into more difficulties.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed