You are correct that the label connectors end at the edges of the layout slot for the bar. Currently there is no way to control how the connector is drawn and where it ends/begins.
Can you give us more information on your scenario, images of the desired results? Perhaps in the future we can introduce some way to make this possible. Or perhaps we may figure out a way how to work-around this.