I am not sure you will be able to achieve this using the standard (text) mask. String values are always left to right aligned so you would need a numeric mask. For the RadMaskedTextBox you can use a mask like n5, allowing 5 digits after the decimal point. To restrict the digits begore the decimal point, you would have to handle the ValueChanging event and Handle it appropriately.
You can also try the RadMaskedNumericInput, with a mask of #4.5. To make it behave and look like the scenario you are trying to achieve, you can set the following properties as well :
Please let me know if you have questions on this.
the Telerik team
Do you want to have your say when we set our development plans?
Do you want to know when a feature you care about is added or when a bug fixed?
Telerik Public Issue Tracking
system and vote to affect the priority of the items