Are you following the approach for creating custom dialogs that is shown in this demo: http://demos.telerik.com/aspnet-ajax/editor/examples/customdialogs/defaultcs.aspx
? I have tested it (including the customized insert link dialog) and it seems to function as expected. As you can see in the code custom dialogs rely on internal functionality in the control which is why they work with passing arguments.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed