It is some kind of interference with: PCB Editor / File / Board Setup / Design Rules / Constraints / Copper to hole clearance. If I set this to a smaller value, then the extra clearance is gone.
It is sort of logical I guess. If you specify a clearance between a hole and copper (from another net), then KiCad takes the biggest of the two clearances, and if you have very small vias then this copper to hole clearance is bigger then the via annular ring plus the track clearance. At least, I think that is what is happening here.