Thanks a lot!
This code you suggested works fine with my test project, but doesn't work in my real application. I found out, that it is character properties of Dispatcher working. This is solution that works for my purposes below. May be it will be helpful for someone else.
RadPropertyGridCollapseGroupBehavior : Behavior<RadPropertyGrid>
AssociatedObject.Loaded += AssociatedObject_Loaded;
AssociatedObject.Loaded -= AssociatedObject_Loaded;
sender, RoutedEventArgs args)
// use DataBind priority to enforce Dispatcher invoke method in appropriate time
Action(() => AssociatedObject.CollapseGroup(GroupName)));