I tried the same scenario at my end and it works as required, i.e., the group collapses even if one row is in EditMode. However i noticed that the edit form does not get cleared when the group is collapsed and if this is the case, you can try out the following code snippet. cs:
if (e.CommandName == "ExpandCollapse")
{
foreach (GridGroupHeaderItem groupHeader in RadGrid1.MasterTableView.GetItems(GridItemType.GroupHeader))
{
GridItem[] children = groupHeader.GetChildItems();