You will have to change the Calendar's template in your theme. If you want to create smaller DatePicker popup you will have to edit the RadCalendar as well as it's buttons and view templates because they have minimum sizes that you will have to override.
I'm attaching an example with DatePicker with small Popup. You can check the template changes in the Resources/Calendar.xaml. I have removed all MinWidth / MinHeight setters. The current height can be found if you search for the words "NOTE:".