Yes, the registration of the dialog handlers in the web.config is mandatory for the RadEditor's dialog to work.
What you can try is to create a physical empty Telerik.Web.UI.DialogHandler.aspx file in the root of the web site and set the DialogHandlerUrl property to point to it.
<telerik:radeditor runat="server" DialogHandlerUrl="~/Telerik.Web.UI.DialogHandler.aspx" ...
If this does not work you should register the handlers in the web.config file.
All the best,
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed