You would set it in the ItemCreated event. It would look something like this:
GridEditableItem && e.Item.IsInEditMode)
GridEditableItem editItem = (GridEditableItem)e.item;
GridTemplateColumnEditor uploadEditor = (GridTemplateColumnEditor)editItem.EditManager.GetColumnEditor(
.....additional code to pull upload control.....
What the code does above is check that the grid in edit mode (raised during insert/update) and that the item is a GridEditableItem (that is the item you'll need in order to access the controls). Now depending on the column type you're using, I'm assuming it's a GridTemplateColumn, but if it's another type you'll need to change the conversion to that column editor type.
From there you'll do a FindControl to pull out your RadUpload control and then set the AllowedFileExtensions property.
I hope that helps.