Note that by design RadMaskedTextBox is supposed to be used when user input should be limited to specific characters only. The control does not fit very well in numeric input validation scenarios. Instead, I would suggest that you utilize the RadNumericTextBox
control which is especially designed to work with numeric values. It provides properties like MinValue(MaxValue), DecimalDigits and NumberFormat which you can successfully use to achieve your goal.
the Telerik team
If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed