I'm not sure that I clearly understand your request, but I believe you've implemented an Add button that is separate from the RadGrid control itself, and you're finding that when you click it the window pops up as expected, however the RadGrid also goes into inline insert mode. So your question is, how do you stop it from going into inline insert mode.
I believe you can achieve the desired functionality by simply canceling the InsertCommand event as it occurs on the RadGrid, like this:
private void RadGrid1_InsertCommand(object source, Telerik.Web.UI.GridCommandEventArgs e)
e.Canceled = true;
If I've misunderstood your question, please accept my apologies and try to explain it to us again.