Since you managed to remove the links from the "Custom Links" dropdown, you are on the right track. To remove an item from the toolbar you have to remove its <tool> node from the ToolsFile.xml.
The Custom Links tool's name is "InsertCustomLink" and The Full-Screen tool's name is "ToggleScreenMode".
For your convenience I have attached a ToolsFile.xml with the default tool set where the above mentioned tools are removed.
More information on the subject is available in the How to set editors toolbars
demo. Also you can find "default" and a "full set tools" files in this article: Default ToolsFile.xml of RadEditor for ASP .NET AJAX
the Telerik team