I have no end of troubles with my parts “housed in a library file”… whether finding them in the first place (by remembering their name … is it 25L8006 or is it MX25L8006EM2I-12G? … is it in the SMD library or in the Memory library?), or modifying them, having duplicates which are different, or deleting… etc
I’m sure we need a scheme to keep parts organized (and secondarily, perhaps a priority ranking scheme), but why must multiple parts be coalesced into a single file?
Its no problem (in Linux) to have 100K+ files in a directory (OK maybe this is extreme) so there’s no need to coalesce for filesystem file number capacity reasons, and no problem nesting directories (using OS utilities, not KiCAD utilities) to EASILY create custom hierarchies, even to the point of putting every part file (logical, footprint, photo, 3D models, datasheets, BOM link to Digikey, etc) in its own directory if desired. From each, according to his needs; To each, his own hierarchy !
I think you have just summed up one part of the (for me) 15 years of frustration with the Kicad library system, A change is coming but it takes a lot of time and effort by people working voluntarily in their own time to make these changes and patience helps to make it a lot easier to deal with.
Of course they don’t have to. However, when you have a lot of parts scalability becomes an issue whether you have one big library or lots of small ones. In software, there is no magic bullet. If you like try creating 100,000 files in one directory, I recommend making a backup first
KiCad has already encountered performance issues with many footprint files, I think those have been fixed for now. Future KiCad will also have separate files for symbols, that might lead to some more issues. I noticed that LibrePCB uses the many files approach but also uses an SQLite database to cache metadata for fast searching.
The new library file format will have one file per symbol. Similarly to how footprints already work.
Originally it was planned to have this in kicad version 5 but because of a lack in manpower, this change has been postponed until the version 6 release.
Just to be clear. Even with the new lib format, the symbols will still live within libraries. (This is a common way of organizing such things in every EDA tool.)
In fact, if everything goes well the user should not notice any difference from this change. At least not as long as the user does only inspect the libs from within kicad. (The new file format will include additional features. But this is a different story.)
I can see some Windows antivirus choking on 100k+ files in a directory. One advantage of libraries is that I can easily choose not to load them. How do you avoid loading the lot?
I wonder how many people would regularly use and have enough symbols & footprints stored on their system for performance to become a problem?
I don’t use any of the libraries packaged with Kicad because 99% of what is in them I don’t need. I prefer to download 1 symbol as needed and 1 footprint if required and place them in a directory/library of my choice and hopefully this will happen with Kicad symbols sometime during my working life
I have already had severe library loading delays on badly fragmented drives
Was that symbol or footprint loading? I did read something about it here a while ago and I do use atomic parts but not enough of them for it to be a problem for me.
Symbols with fragmented library files.
There are industrial KiCad users who already use atomic parts in fairly high numbers and these need to be allowed for
Two different things.
Symbols and footprints stored in my hard disk don’t trigger performace delays.
But (unused) symbols and footprints loaded into a project can potentially slow down the computer.
Only loaded libraries into a project can be used in that project. So, I only load the libraries I need for a specific project. More or less the same as loading the atomic parts needed for a specific project.
Yes, I do see that was a poorly worded statement from me when I meant the number of symbols or footprints stored in each library which I try to keep to the minimum I need
You need to support library files with any number of parts per library. This is especially important when you generate your parts on the fly and all parts reuse the same base code.
You can simply make that number equal to one and you have one part per file. You release your libraries in both forms and let the user decide which one to download onto their system.
You do need to support true hierarchical libraries as deep as needed. No limits on your library depths.
You also need to manage the very top level library name so there are no name collisions. If Digi-key releases a part with the same part name and library name as in the kicad distro then there is no problem because we call out the vendor name along with the library and part name.
This is specified in IP-Xact IEEE-1685. Follow it or forever deal with name collisions.
And you also need to throw a revision number on every part.
Kicad already supports library files with any number of parts per file so kicad already supports one part per file when you build a library with only a single part. Thats what OP was asking for.
Kicad does need to add vendor and version names to the library and part names that you currently use or else you will forever be fighting name collision problems.
Kicad could also release all of its parts in two forms, one part per file and multiple parts per file. Let the users download the version they want.
…just a reminder for everyone and especially the newcomers (you know who you are ).
KiCAD.info is a volunteer support forum and while some developers show up from time to time for various reasons, their playground is the mailing list over at launchpad: https://lists.launchpad.net/kicad-developers/date.html
If you have any time/energy to invest into changing how KiCAD is supposed to work, that will be the forum to use as no one (who does make things happen or can help you make things happen) will notice it here.
Lets not get hung up on 100,000 files per directory… KiCAD currently doesn’t have anywhere near that number of parts.
I agree. Lets not put arbitrary limits, anywhere.
But can we also agree on a duality that not only can a “.lib” file be a library, but also a directory (of parts or .lib files) be considered a library? In other words, intermixing .lib files and .kicad_mod part files into the same (bunch of) directories ( and getting rid of the “.pretty” directories).
Thinking Top-Down, Digikey says they have 6M parts, so a scheme should be able to handle at least that. Creating a main (ie the first major) library directory (say KiCAD-parts.hash), there could be a 1000-way hash (on the .lib or .part/.prt) filename … with 1000 directories called lib.000, lib.001, … lib.999 and Digikeys 6M parts would be distributed evenly (with a decent hash algorithm) with 6,000 parts in each directory (no problem for Linux/macOS/BSD). That is for storing all 6M parts, but realistically, KiCAD currently has (say) 1000 parts? For the current situation, with all 1000 parts hashed into 1000 directories, even Windoze should be able to handle it.
The issue is… how does a library or part (say abcd.lib or abcd.part) get into its correct hashed directory? I suggest putting it into the KiCAD.hash directory (along with the 1000 directories lib.000, lib.001, … lib.999) and next time the part manager is started up, it discovers those loose ends, hashes their names, and places them into their proper hash directory. Hashing is easy and lightening-fast, but if that is a problem, I can provide such a hash-on-the-name algorithm.
How does a human find the file (for whatever purpose)? Just find the file using its name, and the appropriate OS command (ex: find -iname “abcd*” ).
Handling lots of data via the file system is SLOW, and anyone who has tried it knows that. There is a really good reason that databases exist, and it is not to line the pockets of software developers
It’s all quite moot anyway, unless someone wants to start hacking KiCad code, symbols in v6 will likely have the same structure as footprints in v5, ie. a set of “SWEET” S-expression files in a folder which represent a library. I don’t really see a need for anything more complicated.
Let me clarify about arbitrary limits… I had meant to add that other (non-hash, arbitrarily named) directories be included in with the 1000 hash directories, each with arbitrary hierarchy, and that those also be searched (all taking priority over the hashed directories, and amongst themselves - taking an alphabetical priority based on their directory name).
).