The code formatting is done via the IndentHTMLContent content filter and you can disable it with the following line of code:
The changing of the color is due to teh ConvertToXhtml filter, although I cannot suggest disabling it. This filter is intended to provide valid content and when disabled many of the Editor functionaries could fail.
Nevertheless, upgrading with the latest version of the Telerik UI for ASP.NET AJAX suite would resolve the issue. On my end using the same content, the color value is correctly preserved as HEX and not replaced to RGB.
You can do some further testing with this live example
Let me know if you need further information on this matter.
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.