Generally we can't always guarantee that the controls will be compatible with other third-party such. And in this case the transformation provided earlier actually brakes the styling of RadMaskedTextBox.
Currently you have two things that you could do to improve the appearance: disable the embedded styles, and use the old rendering of RadInput that does not show span over the visible Input element
However you will see that in this case there are three visible inputs. And if you like to hide the last two you will need to edit the jquery.jqtransform.js to not transform them. Not sure by it may require a lot of custom scripts.
We are constantly try to improve our products further in order to work better in more and more scenarios. We are working on some improvements of the InputControl and as far as I can test here this example with jQueryTransformation will look correctly. You could wait for the - 22nd of May when Q2 2012 Beta will be available in public.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed