I have sent an answer to the support ticket you have opened about Exporting RadGrid to excel. I would suggest proceeding the communication in the support thread only, and when we reach a resolution, I could share it here also.
Concerning the missing Xlsx in the export format options, I would suggest you go check out and try the suggestions in the Upgrading Troubleshooting section. The issue could be due to referencing the old version of Telerik Control.
Check if the correct version of Telerik controls is referenced - http://somup.com/cYi0226iKR