Turning off VS extensions like JustCode is also discussed in this thread (see Rumen's post from the 1st of March 2012: http://www.telerik.com/forums/not-enough-storage-is-available-to-process-this-command#V33HssR5LUWT3LWApkQV8Q). Since you were still having this issue, I assumed you tried that or you did not use such plugins. The same post treats memory limitations (e.g., using 64bit OS and processes instead of 32 bit).
As for a smaller assembly—at the moment the Telerik.Web.UI is at its smallest possible size. It contains only the code for the controls (over 100, as I already mentioned) and a single skin (Default). This means the embedded resource count is at its absolute minimum so the controls can actually run and splitting the assembly would introduce a breaking change to absolutely all of our customers. The other problem this brings is that there is no universally acceptable logic we could use to split it (for example, should we leave data bound controls in one assembly, layout controls in another, input controls in a third one, and the common code shared across all controls in a fourth; or how do we determine the most used ones, apart from RadGrid).
I hope this clarifies the situation.
UI for ASP.NET AJAX