There are a few different templates where you can modify the style of the close button. Please have a look at the following article to get an understanding of how to extract and edit the templates.
To change the style of the button inside the DocumentHost, you need to edit the RadPaneGroupDocumentHostControlTemplate.
To style the buttons inside the pane headers outside the DocumentHost, you should modify the PaneHeaderControlTemplate and for ToolWindows - the ToolWindowControlTemplate.
For your convenience, I've attached a small sample project demonstrating the required changes for the Material theme. Please note that if you're using another theme you should modify its templates accordingly.
Please let me know if you find this helpful or if you require any further assistance in the process.
Get quickly onboarded and successful with your Telerik and/or Kendo UI products with the Virtual Classroom free technical training, available to all active customers.Learn More.