To achieve the desired functionality you need to implement your own expander cell and use it in RadGridView. Please find attached a sample project, demonstrating how to implement group expander with checkbox. Note that this is just a sample approach and it may not cover all cases. Feel free to modify it according to your exact requirement. The implemented functionality is illustrated on the attached gif file.
I hope this information helps. Should you have further questions, I would be glad to help.
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.