I'm trying to fetch a lot of data, which is reaching 100k+ results. The main problem is that the API doesn't work with this many records at once due to simply being more data than allowed in memory by the webserver (for good reasons). What's the proper solution to continue here? I can easily page the data, but I want the filter/sorting client side over the complete result. It basically comes down to having to execute multiple requests to fetch all data and then use the grid with this data.
I noticed several issues if I use a DataSource directly, such as paging will request a new page if the page size changes for example. If I have already fetched the first 20 records, and I change the page size from 10 to 15, it will fetch the 15 records again while it already has those records. On top of that it seems that I can't exactly pre-load the data easily either without manually writing some code to call the API endpoint X times.
I'm suspecting that the solution would be to create an observable array and fill this based on a data source that I query page by page? This way I could for example load the first 100 results and then batch the rest in sets of 10k or 20k?
Are there any examples available? I'm using the jQuery variant.
Edit: I'm still in need of at least a "destroy" call for this grid, could that even work together?