Any way to change default font size for symbols?

That’s probably the way to go. Standard (ISO, IEC, etc.) size would be 2.50 mm (≈ 2.54 mm = 0.1″ = 100 mil) anyway (i. e. grid size, which would be 1.8 mm or 2.0 mm minimum). Smaller would require to scale down everything (fonts and symbols). Example here. (Before computers, a common schematic paper size was ISO A3 landscape, which was then scaled by √½ ≈ 70.7% to ISO A4 if required.)