Make sure Dialogs are enabled - http://docs.telerik.com/devtools/aspnet-ajax/controls/editor/functionality/dialogs/overview
If that does not help, then you should investigate what causes the DialogHandler.aspx from serving content. My advise is to open a support ticket with enough information for reproducing the problem.
Hope this helps.
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Feedback Portal
and vote to affect the priority of the items