We strongly recommend not to export large amounts of data since there is a chance to encounter an exception (Timeout or OutOfMemory as it is in your case). RadGrid is not suitable for such scenarios and therefore we suggest that you limit the number of columns and rows.
However, you could try using Telerik document processing Libraries
. Thanks to these libraries you can directly traverse your datasource and generate the XLSX file which supports more than 65,000 rows unlike the BIFF format which is an older and more limited one. You can examine our online documentation which would help you get started easily.
RadSpreadProcessing Getting Started
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.