Currently RadRichTextBox takes all space provided by its container when the layout mode of its document is flow.
We are currently working on the behavior that is expected when measured with infinity (what happens in a Grid row with Auto height or in a StackPanel). Note that in these cases, RadRichTextBox will not be able to make use of UI Virtualization and that will most probably have a negative impact on the performance.