I see what happened. I had initially stated this wrong. The binaries are not in the GAC normally so that is a bad statement. However, what I observed was that Telerik controls were not showing up in the references under .NET tab as they usually do, and this was breaking my build.
The problem appears to be that our project setting was still set at Silverlight 4, and the references tab is filtered to the applicable silverlight version. So after updating the setting to Silverlight 5, the tab is filtered to Silverlight 5, and the Telerik controls show up there as expected. So just be aware of this if you have not moved your project to SL5 yet!