Question: How important is it for you to have the reference there on the board for hand assembly?
See those threads for how others solve this ‘problem’:
There was a thread here from a regular who wrote a script that was able to adjust the reference values in pcbnew from their position on the pcb (similar to what eeschema does in regards to numbering vs position on the schematic sheet) and this was then back-ex/imported into eeschema, so the link stayed intact.
Might not help you in it’s current form, but maybe you can reuse some of the code?
PS: why do you need reference AND value in silkscreen on the board? Isn’t that a bit overkill?