I converted existing ajax ModalPopupExtender window to rad window. The existing page had a ajax felexbox(jquery) ,after i convert to rad window it disappears.
I examined the FlexBox plugin and ran it inside a RadWindow template (which I believe is your scenario). The problem that I see is that the FlexBox plugin works in Quirks mode only. If there is a XHTML DOCTYPE on the page, the dropdown is displayed outside of the browser's viewable area (see attached screenshot). That is the same behavior as the one that I experienced when I put the plugin inside RadWindow.
Please note that RadControls for ASP.NET AJAX (including RadWindow) are built upon the MS AJAX framework that uses <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
and our controls rely on that DOCTYPE to properly position and display themselves on the page. If this is the problem that you experience (i.e. the dropdown is not visible), then I would suggest to contact the plugin developer for more information on how to make FlexBox working in XHTML-compliant mode.