As far as I know for KiCad V4.0.7 the used Footprints are stored in the .kicad_pcb file itself, and that’s why you don’t loose them.
Rene has written an extensive FAQ entry about library management, converting & migrating from KiCad V4 to V5.
https://forum.kicad.info/c/faq
The most interesting FAQ entries for you are probably:
https://forum.kicad.info/t/library-management-in-kicad-version-5/14636