Not quite. I think it’s actually a term for people who download zip files to RAM. 
1 Like
I wanted to download Git zip files to magnetic hd sectors…
As an experiment, I updated Nightly on my Windows 7 64 bit PC, 36% disk used. The disk was defragmented clean before the update. After the update I had thousands of fragmented files in the KiCad folders. Windows is doing a poor job of avoiding fragmentation and KiCad is especially sensitive.
Using the file system as a cheap database is tempting, but it is very inefficient. I’ve seen it tried a few times, always fails with scalability issues. Having lot of files with a few kB in each is the worst way to do it. The major overhead is in opening/closing the file, the read time is negligible.
It would be much better to store all the footprints for a library in a single file. That would also make it easy to put an index at the start of the file for quicker access, searching.
So… back to how it was before kicad 4.x? (And how it still is for symbols)
Having it in one file per symbol/footprint is a lot nicer for us library maintainers though. (Less merging issues.) Not sure what exactly the pros and cons are.
Then do both. Keep individual files in the repository but then run an app or script that builds the “library” by merging them into a single (or several, but not hundreds) indexed file(s) for release.
To be honest I don’t understand why they changed it in the first place.
Complaining that there is no possible reason for something to be implemented a certain way is a favorite pastime of people who have not seen the usually large number of alternatives and reasons that were considered when the decision was made.
Not understanding the reasoning for a decision is not suggesting “there is no possible reason for something to be implemented a certain way”. But why don’t you enlighten us?
To be honest, I don’t know either, I just think it’d be valuable to look into why it was done a certain way first and not assume we hadn’t considered other options.
Not trying to be snarky and dismissive. Honestly I also think having everything compiled into a single library file would be nice, though separate files are also nice for other reasons. I’m more frustrated than anything - in many cases I totally agree with users’ complaints about KiCad. The entire file format/parser/writer situation is a complete mess. In fact it’s a poorly designed mess meant to replace another mess, that just resulted in two messes. It gets tiring to feel as though you’re constantly on the receiving end of complaints you can’t do much about, though. I’d rip out huge parts of KiCad and totally reimplement them if every one of them didn’t have someone behind it who thinks it’s the greatest thing since sliced bread, and if I didn’t also have a full time job where I can’t work on them.
I’m sorry if that frustration came out at you.
2 Likes
Please all keep polite and constructive here.
I can see why individual files can be preferred. A separate index file with part names and some kind of versioning for each footprint would solve many of the performance issues
1 Like
The move to .sweet symbols will compound file loading delays. I would look seriously at using a local sqlite database to manage these thousands of files. searching this database is quick and tasks like duplicate detection become easy.
Assigning version or timestamp allows the local installation to only download new or updated files prompted as required to avoid breaking a project. This would make Nightly updates a sensible size again. I noticed that these jumped by 100MB on 16/7/17 to 670MB, presumably .step files getting merged.
2 Likes
I’m really unhappy with the direction file load efficiency is going in.
Also, I think versioning has been proposed, and did not receive a warm welcome. I get around this issue by using git submodules to give each project its own frozen version of my libraries. I wish I could build something into KiCad itself to take care of this…
1 Like
Some of the consequences of lack of versioning/timestamp/hash is that all files have to be updated just in case they have been updated, projects suddenly break because of a silent update and we have to download almost 700MB every day of the new Nightly.
Starting KiCad on a PC with a fragmented conventional hard disk could take five minutes with .sweet and .step files to be opened.
1 Like
Yeah, I’m totally with you on versioning.
1 Like
Most programs check to see if there is an update.
Possibly check to see if the LIB has an update, and if that part has been used?
Optional to update the library, and optional to “rescue” the part used in the design (both Eeeschema and PcbNew).
The “rescue” would move the “older used part” into the project itself.
Maybe add a new library with “old parts” for the user?
I see where this library “stuff” seems to be headed, and I want no part of this trainwreck.
The PCBNew Footprint Library Wizard does not check to see if a library is stale.
If a GitHub hosted library is selected, the entire directory is updated (slowly), not just the changed file.
This is not efficient
1 Like
@davidsrsb “not efficient” is the most generous thing you could say about it. I have been fighting tooth and nail to nix the GitHub “plugin” approach, but there are a lot of people wedded to it as @c4757p has alluded.
The only way we’re going to improve the situation is to develop a clear workable solution that completely eliminates the use-case of the current plugin. As long as people are still loading live libraries by (and I wish I was joking) - downloading a zip file via a non-Git GitHub API (that may change) and then caching those in RAM - then the official developer position is that these users must be supported.
As with @c4757p I would love to see this improved, especially as the person currently holding the poisoned chalice of “Library Administrator”.
1 Like
It is STUPID. <— yes caps.
Why am I not given the option to choose to not download libraries that have NOT changed?
I think this topic has made my brain hurt.
Use the full opt-out method: download the libraries to your own system yourself and don’t use KiCad’s GitHub feature. I think it’s asinine, haven’t ever used it beyond testing changes to it. I manage my own libs so I keep them all in one repository, but there are tools for managing sets of git repositories as well - you could look into those, as they can handle going through and downloading updates.
My advice to almost all users is to just pretend the GitHub plugin doesn’t exist. KiCad was plenty usable before it was added.
2 Likes
methinks that “full opt-out method” button is not something on the menu bar that I can click…