@Mantvis, first of all, thank you so much for your detailed posting!
I am confused about one thing. In this earlier posting (granted, for KiCAD7; now we are on KiCAD9!), @teletypeguy extensively discusses a JSON file which translates between KiCAD and the database. @craftyjon also discussed this file in his presentation at KiCon2023 in Europe (advance the video to 7:30). I don’t see any reference to such file in your posting. Has something changed from KiCAD7?