Actually, RadGridView with GroupRenderMode=Nested would appear in the same way as RadGridView with GroupRenderMode=Flat. I will try to explain what are the differences between the two modes:
The old mode (Nested) would nest GridViewGroupRows into one another when you had grouping on many levels. This lead to a very poor performance when the grid was grouped on several levels and had a lot of data. The entire groups had to be realized instead of realizing just what is seen on the screen. So if the groups were very high -- virtualization was almost useless and it was slow.
On the other hand, the new Flat mode simply renders rows one below the other -- it does not nest anything in anything. This leads to a huge performance boost when the grid is grouped on several levels and there is a lot of data involved.
Thank you for the suggestion about our documentation. It will be updated next week.
All the best,
the Telerik team