Just to give you more information on the issue you have faced - I tested your application scenario with WCF RIA Services for SilverLight 4.0 and Visua Studio 2010, and RadGrid works all right. I am also attaching a small sample that should serve you as a guideline.
One thing that I suspect might by the reason for the issue on your side is the fact that besides the QueryParameters for the data source of the nested table views you need to specify also a QueryName (i.e. a select method) that takes into account those parameters.
Do take a look at the attached sample at how I have implemented this and let us know how it goes.
the Telerik team