This sounds just like this memory leak, which Microsoft have recently fixed. In case upgrading to the latest version of WCF RIA Services does not resolve the memory leak, can you please see what objects are leaking with the help of a profiling tool or WinDbg and send us the results.
By the way, have you considered using our new RadDomainDataSource instead of the stock MS DomainDataSource?
To explore the new RadDomainDataSource control in greater depth, please check its online examples and take a look at the following blog posts:
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items