Pcbnew / Preferences / Preferences / Pcbnew / Magnetic Points / Snap to pads
[Edit] Oops wrong function indeed. The link above adjust whether you want to use a pad as anreference point to snap other things to. Not which location is used as a reference for moving a Footprint itself.
But I’m not sure if you want to turn it off.
When you grab a Footprint by a pad, then the center of the pad is used as an anchor point, however, if you start with Silkscreen or something on the Fab layer, then the center of the Footprint is used.
A thing that I do find annoying is that the old location of a Footprint is used as an additional snap point, which makes it difficult to move a Footprint a little bit. It’s also not needed. If you want to back off from a move you can simply undo it.