The Kendo UI Grid doesn't support the ability to use both inline and pop-up edit modes at the same time. I encourage you to participate in the Kendo UI Feedback and Feature Request Portal
in order to voice your opinion and, if enough people vote on it, it might get included by developers in future releases.
That said, you explain a custom approach that should work in your case; I would recommend using inline
as the default edit mode and using a custom command button that opens a generic Kendo UI Window with your custom edit form. When the form is submitted, if the data is valid, insert the newly created dataItem into the DataSource and execute the grid's saveChanges method.
Keep in mind that this would be a custom implementation and thus might not be fully supported or compatible with other implementations or future versions.
I hope this helps!
Telerik by Progress