You will need to edit the ControlTemplate of the RadDropDownButton (read more about editing ControlTemplates in Blend) and change the Background color of the DropDownPopupBorder element.
By default the DropDownPopupBorder Background property is bound to a SolidColorBrush resource named ControlBackground so if you change this brush's Color to Transparent, the DropDownContent Background will be transparent.
I attached a sample project to get you started. Let us know if we can further assist you.