This is a two part issue and should see improvements to the defaults along with interactions for font sizing. There are a few elements to this design and 2 variations:

• Change the A icon to mean something more 'font size' for example A+ or different sizes of A.
• Remove the slider and in an editable field and arrow up/down.
• Add in default value that is what the font size is for that block.

Why? This makes adding easier, more accurate and should stop the skip/sliding as you try and get the right font size. Also by showing the default it means you have some baseline.