Brian, this tracks with my suspicions.
I had dabbled with DIV and other local style variations, but with the same result.
After reading your post, I removed the LI styles and the problem has dissappeared.
My current working hypothesis is that the Editor is unusually susceptible to its working environment.
I can live without a global LI style, as I used UL/LI as a crutch to avoid investing in learning CSS until I was sure that the Internet would be a success. It is <g>
All kidding aside, the few pages that absolutely need UL/LI are not the ones that will have RADEditor, so I think you have solved at least my problem.
As for yours, it maybe be helpful to know that I did find I could retain a simple LI that did not include margin or any other layout measurements. If this holds for you, then I would amend the thread title to "Improper toolbar spacing due to global UL/LI styles"