You are correct using the Document Manager, ESC key closes the whole dialog. This is due to the same matter described in point 2.
As suggested using a Customized Dialog you should get the LinkManager.aspx file where the suggested code can be found and modified. Following the steps provided in the online demo should guide you on how to customize the dialog.
If you have any difficulties, please provide exact details on what exactly is troubling you on following the provided steps in the Customize Built-in Dialogs
As already mentioned, such matters are designed to be optimized and fast, and correct upon user experience. The suggested behavior is not possible due to the above matters.
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.