The version you are using is almost three years old and IE9 was not around when it was created. The first version that supports IE9 is Q1 2011, yet I advise that you upgrade to the latest version which is currently Q2 2012 SP1. You can also check this thread
for a temporary override that may enable the RadWindows to work under IE9 with your version until you upgrade. You can also examine the x-ua compatible meta tag
to get IE9 to use IE8 mode.
Regarding the error message related to the browser add-on - the error IE9 causes for older versions of the RadWIndow is a DOM error 5 and not this browser message. This means that it is possible that some content (e.g. a Java applet, a flash object, a PDF) requires some plugin that is broken on the browser you use, so I would advise that you check this as well, because it may not be related to the RadControls, but to the browser and the page's content.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed