Process Monitor shows that reading the .libs takes much less than a second. The delay is due to reading every footprint, one file at a time
Uh, it actually parses each footprint file? That seems like an oversight - it should only parse a footprint when it comes time to display it, I would think. 99.9% of the time a particular footprint won’t get displayed in that window.
You should definitely submit a bug report for that
1 Like
Nightly has a footprint preview. Time for KiCad to stop loading every obscure library by default
Kicad stable does load very few schematic libs currently. (I think only 30 or so, but nearly 100 are installed. Might depend on operating system/ who packaged kicad.)
Follow the forum a bit and you see a lot of complaints about that because it is quite hard to get kicad to add a specific lib to each new project. (You need admin or sudo rights to change the template.)
As usual, there seems to be quite a bit of confusion here between component and footprint libraries. I have also found on 4.0.6 that the first call to cvpcb in a session is quite slow (hangs for maybe 30 seconds) while the footprints are loaded to memory cache. I am using local footprint libraries on main HD
Loading all (~100) component libraries is quite quick (2-3 seconds), that is not an issue. KiCad defaults to a small selection of component libraries, there would be no reason to add all of them by default.
Loading all footprint libraries is slow. Not sure why this is the case, I think it is probably slow parsing in KiCad.
Virus scanners normally kick in when files are written/closed, so that could be an issue. Eagle keeps all it’s libraries in text XML files, even with many libraries I’ve never noticed significant delays.
The priorities should be
- make sure parsing is optimal. Often buffering IO to reduce OS calls is all that is needed.
- implement demand loading of the memory cache
- use a background thread to fill cache while user does other stuff.
2 Likes
4.0.6 only reads footprints when you open PCB, Nightly appears to do it in Schematic when the add component icon is selected.
As each footprint is a file, opening all footprints is slow in Windows, antivirus making things even worse. Only loading a .pretty directory when required by selecting a symbol library in the component chooser would avoiding blocking for minutes. Background loading of the rest would help. KiCad is basically single threaded. Having a separate thread for reading these files would be a good idea.
1 Like
Or if you run cvpcb? Whichever bit of code runs first that needs the footprint cache causes the initial cache fill. It sounds like that has been brought forward.
The interesting question is, is the footprint loading actually slower, or is it just being done earlier? Are the 3d models also being loaded?
1 Like
Aren’t footprints stored in the pcb file once they are “imported” (for lack of a better word)? I think access to the lib only happens if you import a netlist (with new footprints), if you use the change footprint feature or add a footprint manually.
The Launchpad report is closed as this is expected (bad) behaviour, loading all footprints.
I have found a considerable improvement to 50 seconds by defragmenting the disk. Installing Nightly builds regularly has had the effect of scattering the footprint files.
This expected (bad) behavior supports a feature that’s not fully implemented yet, so it may seem like bad behavior still.
I very strongly would like to get all of the footprint loading fully backgrounded - it’s partially done, but due to some truly eldritch horrors in the parsing code there are sections that must run in the foreground thread, freezing the GUI. If I have a lot of time on my hands and feel unusually masochistic sometime between now and the 5.0 release, I’ll see if I can fix the parser.
Anyone who wants to get your hands dirty and is experienced with very, very bad C++ code and issues having to do with locale and encodings, I will buy you several good beers to submit a patch removing LOCALE_IO…
If I have to, I’ll just make the footprints no longer “lazily” loaded so they load at startup - if there’s going to be a delay somewhere it’ll be a lot less annoying at the very beginning than when you’re just starting to actually dig in. But I’d be seriously unhappy with that. It’s a long processing step and really, really should run in the background. I’ve built the framework to make it totally backgrounded, it’ll even keep running after you close the component chooser - just have to deal with LOCALE_IO… 
2 Likes
Currently, this is the only way to actually get a list of all the footprints, which is necessary to allow footprint selection. If I have to, I could write a lightweight parser that just extracts the names, but that’s yet another bothersome half-solution.
Side note: we really should have a format for separate footprint name listings - normally I’d be against that as it’s duplication of data, but since we allow footprints to be downloaded directly from the internet, being able to quickly fetch a list first would be valuable…
1 Like
A list of the names of items in a library is (to me) separate from the actual items. I think this would be valuable. Just to be clear, I imagine that in the component selector, a footprint is not read (cached) until it is required to be displayed. It may mean a little bit of lag each time you select a part, but unless the entire lib-read is performed in a background thread I think this would be the optimum solution?
The component selector doesn’t just display footprints, it lets you select them. That requires knowledge of which ones are available. Unfortunately there is no guarantee that footprint file names correspond directly to footprint names, so collecting such a list at this time requires actual parsing.
Some sort of index file with footprint date or checksum signature, would not just help this issue, but could also solve the problem that I frequently update a footprint library (or package3D) from GitHub, only to find that there was no change.
This is a terrible waste of GitHubs bandwidth
I’m still waiting for the assassin sent by GitHub to show up.
If only we could work out what the “Git” part of GitHub meant. Does it mean “download zip files to RAM”?
1 Like
Not quite. I think it’s actually a term for people who download zip files to RAM. 
1 Like
I wanted to download Git zip files to magnetic hd sectors…
As an experiment, I updated Nightly on my Windows 7 64 bit PC, 36% disk used. The disk was defragmented clean before the update. After the update I had thousands of fragmented files in the KiCad folders. Windows is doing a poor job of avoiding fragmentation and KiCad is especially sensitive.
Using the file system as a cheap database is tempting, but it is very inefficient. I’ve seen it tried a few times, always fails with scalability issues. Having lot of files with a few kB in each is the worst way to do it. The major overhead is in opening/closing the file, the read time is negligible.
It would be much better to store all the footprints for a library in a single file. That would also make it easy to put an index at the start of the file for quicker access, searching.