In radgrid if I click on edit link it has to open a new window in browser (as per browser settings it may be new window or new tab in current browser window) edit page, and new page has to load my user control. What settings need to be done in UI....
If I understand your requirement correctly, I'd suggest that you reference the Grid/Window Editing online demo. When the "Edit" link in a row is clicked, a new window pops up that contains all of the controls required for editing that row.