It seems like "exception of type 'system.outofmemoryexception' was thrown" is a common error that people are having using the RadGrid; however, I can not find any real solution that works on the forum. My dataset contains 3.5 million records for 80 rows of data.
This might happens because of large record set.The following help article describes the methods to improve the performance of RadGrid. Client/server grid performance optimizations