Thanks for the reply Martin.
I had tried both of those. The smart labels mode is applied to the axes. The labels that were killing my performance were series labels.
I tried all rendering modes and had no difference.
What *did* alleviate the issue is to remove this code from my chart:
My chart went from 40 sec down to 5 sec. Still not great but much better.