Home / Click-drag-scroll without selecting text?

The ability to click-drag to scroll the page is very convenient. But I find it annoying that I often select text, thus preventing the scroll action … Can I disable or circumvent that somehow?

You can circumvent by using right mouse button (though natural tendency is to use left mouse button)

After a while I got used to only picking white space for scroll (admittedly that is difficult when zoomed in and most of the screen is body text)

AFAIK there is no way to disable selection of text when you are within it.