We discovered that in the latest version the RadWindow close function is no longer working when oWnd.maximize() is used.
This happens independently which skin is used.
This is a known issue in R1 2017 and we intend to address it for R1 2017 SP1. You can find a list with similar issues in the following sticky thread: http://www.telerik.com/forums/known-issues-and-important-changes#GQBgODtL3EeNY-oLg5qt0g.
I hope the provided workarounds will be useful for you: https://feedback.telerik.com/Project/108/Feedback/Details/211718-radwindow-throws-error-when-maximized-via-js-but-behaviors-does-not-contain-maxim. Otherwise, you can revert to R3 2016 SP2 and wait for the upcoming service pack.