To enable the dialog in custom WebParts consisting of the RadEditor control, you should properly configure the appropriate handlers, as explained in the KB. If you still have issues, I suggest verifying the correctness of the web.config registrations and that they match the correct version of the fully qualified names.
Note, that the RadEditor WebPart is built as a custom WebPart. Due to that I do not see what is the difference if a custom one is used or t he already built and provided. Adding it to a SharePoint page is simple as the custom WebParts. Ones installed and enabled, you need only to add it as a WebPart in the possible placeholders.
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed