The RadWindow does not influence the Session or ViewState (well, it stores its server-side properties in the viewstate, but this is a minimal amount of data). What may be happening is that your ViewState gets too big or there is some other error thrown. I can suggest you examine this article
on optimising it for the RadGrid.
Also, since the RadWindow is essentially an iframe it behaves like one. And there are some issues under IE with iframes and cookies (see here
for example, or try a generic google search
on the matter).
I hope you will be able to resolve this issue. When this happens we would be glad if you share the origin and fix for this with the community in case somebody else stumbles upon a similar problem.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed