In the attached gif, the buttons inside the RadWindow are custom one which comes from the DialogView class. Correct me if I am wrong. As these buttons are custom, I have only tested the "x" button in the top right corner. When the button is clicked, the window is closed. You can find the sample project attached here. May I ask you to take a look at this project and let me know what else I am missing from your implementation to reproduce this. If it is will be easier for you, you can modify the project and send it back to me.
Progress is here for your business, like always. Read more about the measures we are taking to ensure business continuity and help fight the COVID-19 pandemic. Our thoughts here at Progress are with those affected by the outbreak.