Steps to reproduce:
1. Click image icon
2. Resize the modal window lager than the window content area
3. Resize the modal window lager than the window content area
I have also attached screen shots capturing the formatting issues.
The first image is the default view. The next 2 images show the formatting errors.
It would appear the simplest option is to set resizable: false for the modal window. The hyperlink window does not allow for resizing. So it would make sense this modal window would not be resizable either. The other option is to have the image browser content expand and contract with the window.