Thank you for writing.
You should use a small image with transparent background that will fit in the RadApplicationMenuButton. Please refer to the attached image that follows these requirements.
I hope this helps. Do not hesitate to contact us if you have other questions.
the Telerik team