I am not completely sure what do you mean by your own map server, however, please consider the following. Most of the map providers which RadMap supports (BingProvider / OpenStreetProvider) inherit from TiledProvider class which uses TiledMapSource (TMS) class. The TMS has GetTile method which returns Uris to the Images which are downloaded / accessed locally (or from service) and then displayed in the RadMap.
So an approach that seems applicable for your scenario is creating custom Map Provider. For more information regarding it, please check this help article:
Implement Custom Map Provider
Telerik by Progress
Do you need help with upgrading your AJAX, WPF or WinForms project? Check the Telerik API Analyzer
and share your thoughts.