This is given by your display. Also in most cases it is easier to have low side switches because than you can use n channel fets in common source “mode”. (I don’t know a better word here.)
If i remember your circuit correctly you have 4 high side switches to select the digit and 7 low side switches to select what led’s of this digit should be on.
As the low side switches are not part of your pcb but are on the pcb your pcb connects too there is no common gnd.
There is only a common v+ (the common line of your high side switches.)
Here a simplified version of what you pcb looks like. (not all led’s of a 7 segment digit shown, resistors removed, not all digits shown, gate connections not shown)
This might make it more clear.
