As I noted in my previous reply, the implementation of a custom format provider is not a trivial task.
I could only suggest that if you are going to implement your own format provider you would need to implement IDocumentFormatProvider interface. If you wish the open command to recognize your custom format provider extension (.dotx in your case), you should register it like this:
Also I would recommend you download the source code of the controls and use the OpenXml format provider as a base point for your custom format provider. The files related to the implementation of DocxFormatProvider are located at ..\Documents\Flow\FormatProviders\OpenXml\Docx.
If you do not want to implement the provider from scratch you could try implementing the interface and using the default DocxFormatProvider in the Import() method of your provider. This would allow you to open files with extension .dotx, but again you should be aware that some discrepancies in the data might be observed.
I hope that this information would be helpful for you.
DevCraft Q1'14 is here! Watch the online conference to see how this release solves your top-5 .NET challenges. Watch on demand now.