I'm not sure that I clearly understand your request, but I believe you've implemented an Add button that is separate from the RadGrid control itself, and you're finding that when you click it the window pops up as expected, however the RadGrid also goes into inline insert mode. So your question is, how do you stop it from going into inline insert mode.
I believe you can achieve the desired functionality by simply canceling the InsertCommand event as it occurs on the RadGrid, like this: