I do have an SSD, and both my OS and all programs are installed on it.
My KiCad projects are not however, those are on my data HDD.
I should have mentioned that this only / mostly ? happens when the fp-info-cache file is re-generated, but I had times when it seemed to always happen when I switched between projects.
Recently I had to switch from KiCad Stable V5 to the JS-Reynaud-nightly V5 for KiCad V5.1.x because of a library conflict in OCC between KiCad V5.1.x and KiCad V5.99. and now I can not reproduce it anymore. This newer nightly is a lot quicker in library loading.
The JS-Reynaud version does not create the fp-info-cache file anymore in the project directory. And when I delete it, it just stays away.
Adding the first schematic symbol in Eeschema now takes approx 5s, while adding further (other) schematic symbols are near instantly.
The link below suggest that KiCad V5.1.11 is close to release: