Are you using the pan function that activates when pushing against the side of the Eeschema drawing area?
Simplest solution is to get used to it, just pump briefly into the side instead of a long push.
Pan speed in V5.1.9 seems usable to me, but because I personally do not use that function I do not know of scroll speed changes. Instead I make a selection, and then while dragging use the scroll wheel to zoom out and then in again on the area of interest. This way you have both a better overview and more control over locations.
This works especially nice with the default setting of Preferences / Preferences / Common / Pan and Zoom / [x] Center and warp cursor on zoom. Some people turn this feature off quickly after starting to use KiCad, which is quite a shame, because if you’ve given yourself a bit of time to get used to it, you would not want to live without it.