There is a workaround for playing video encoded with Expression encoder 4 in a RadMediaPlayer. I have been able to implement it and it works as expected.
There is an alternative suggested in this thread: http://www.telerik.com/community/forums/silverlight/media-player/failed-to-load-media.aspx#1594404
if you are after the RadMediaPlayer to play Encoder 4 output, then follow these steps: (it involves customising RadMediaPlayer code)
RadMediaPlayer has a media element at the heart of the control.
Replace the media element with Microsoft.Web.Media.SmoothStreaming.
Modify the RadMediaPlayer.
GetTemplatedChildren method to get the control and set it to the MediaElement Property.
You can even switch between the original MediaElement and SmoothStreamingMediaElement based on IsStreamingSource property.
The downside is that you will not be able to upgrade to the interim hotfix releases from Telerik. Upgrading to the next version generally involves re-applying the customisations to Telerik's code. In the end it works.