When using RadWindow for ASP.NET AJAX you can not set the callback function client-side as this functionality is available with RadWindow for ASP.NET. Instead of the callback functionalit, you should use the closing methods. You can find more information about this here
If you need further assistance, please, open a new support ticket and send us a small, working project with a detailed explanation of the desired behavior and the exact scenario. Once we receive it, we will do our best to help.
the Telerik team