As usual, there seems to be quite a bit of confusion here between component and footprint libraries. I have also found on 4.0.6 that the first call to cvpcb in a session is quite slow (hangs for maybe 30 seconds) while the footprints are loaded to memory cache. I am using local footprint libraries on main HD
Loading all (~100) component libraries is quite quick (2-3 seconds), that is not an issue. KiCad defaults to a small selection of component libraries, there would be no reason to add all of them by default.
Loading all footprint libraries is slow. Not sure why this is the case, I think it is probably slow parsing in KiCad.
Virus scanners normally kick in when files are written/closed, so that could be an issue. Eagle keeps all it's libraries in text XML files, even with many libraries I've never noticed significant delays.
The priorities should be
1) make sure parsing is optimal. Often buffering IO to reduce OS calls is all that is needed.
2) implement demand loading of the memory cache
3) use a background thread to fill cache while user does other stuff.