Note that the code will resize the RadWindow dialog only but not the contents inside it because they have fixed size. If you want to customize the appearance of the inner content inside the dialogs you can see how to load the external dialog files of RadEditor in this live demo: Customize Built-in Dialogs.
The contents inside the dialog are located at \EditorDialogs\FileBrowser.ascx and you can control them via CSS.
For example to move the Insert button add:
<style>
.reLightweightDialog.RadForm.redImageManager body {
Get quickly onboarded and successful with your Telerik and/or Kendo UI products with the Virtual Classroom free technical training, available to all active customers.Learn More.