For some time ago I started to make separate KiCad global library subset of parts used in our projects. Our little (already not so little) subset library has parts missing in KiCad global library, parts where manufacturer provided symbols are more precise or than ones in KiCad library (or exactly opposite), parts with missing footprints and 3D models (especially these, because it isn’t possible for user to correct existing footprints in system directories due to filesystem permissions) and so on. Ordinary stuff. All named by using KiCad library naming convention and structured by directories that is named in same way as KiCad library, except our subset has “_EXT” suffix added to component group folders.
Now the question is about the possibility to migrate our global library changes together with KiCad configuration update. Updated component files in Git repository are relatively easy to move with working network access. However updates in fp-lib-table and sym-lib-table require editing these files manually. Not the nicest task considering these file locations in hidden configuration directories, directory naming conventions between operating systems and differences between our KiCad library directory location in actual user system. Maybe there already is an utility or KiCad plugin capable to synchronize user changes in global library? Even better if it would be also capable to export all user added library subset to ZIP archive together with these library definitions in KiCad configuration and restore it in different system?
The problem of importing user provided library entries when upgrading to newer KiCad versions is a known one and hopefully will be solved in a future KiCad version.
Differences across OSes and storage locations should be solved using the same variable substitution mechanism used by the official libraries. For example I define the variables KICAD_USER_SYMBOL_DIR and KICAD_USER_FOOTPRINT_DIR to be the top directory of my personal libraries and use them in table entries. Thus I can move my libraries on my system and all I have to change are those definitions. I only have one OS, but it also works to absorb OS path differences.
Yes, I know about KICAD_USER_* directories and I’m using them now. Unfortunately they alone can’t help to accomplish seamless synchronization between systems because fp-lib-table and sym-lib-table still require manual update when new footprint and 3D model group subdirectories are added.
Perhaps I’m misunderstanding the question, but AFAIK fp-lib-table and sym-lib-table are rebuilt first time you use the libraries in the editors. That’s why you see the process bar.
File date/time supports my understanding.
It happens with libraries in project scope, but not with global libraries. Also KiCad doesn’t support two libraries with same name nor add prefixes to “duplicates” automatically. And I don’t want to add extra prefixes to library subdirectories to keep their relation to KiCad default global libraries visible for users.
I do not consider editing the fp-lib-table and sym-lib-table files a big issue. It’s a few lines in simple readable text files, and you do this once a year for an update of a mayor KiCad version. I consider it just a normal part of library maintenance.
Sure, while you didn’t started to edit user global libraries. Because not always all symbols or footprints are in state when they’re ready for committing into Git, it require to move everything changed together with updated fp-lib-table and sym-lib-table files when I want to go to home and continue to edit libraries here. It become pretty annoying when some added subdirectories are forgotten by accident.
I think there’s a semantic problem here.
To me, all libraries are “global”, unless they’re specifically defined as “project” libraries (which I think is discouraged in 8.0?).
I operate with “standard” or “default” libraries, which are those supplied in the KiCAD install package, and “personal” or “custom” libraries which are user/organisation defined.
But both generate a new sym-lib file when invoked (in user space). Editing that would be futile.
But both generate a new sym-lib file when invoked (in user space). Editing that would be futile.
Unfortunately not always. Only directories defined by few hardcoded environment variables are used in automated library configuration update. The rest of directories in “Configure Paths” dialog directory list are ignored. My libraries are located under /home/<user>/KiCad directory in directory structure
Do you think automated global library refresh take my directories in account? No. Because they aren’t specified with hardcoded directory environment variable or whatever KiCad they specify internally.
You need to be more specific. Which refresh are you referring to?
Additions of symbols and footprints caused by package updates to the standard libraries do not require any change to the table files. No new libraries will be added during the lifetime of a version.
Upgrades to the KiCad version will disrupt the table files. This is a known problem.
Adding user libraries does require modifying the table files which can be done with a text editor (when KiCad is not running) or through the GUI. If you are managing a group of accounts this can be a pain. You could also consider implementing group libraries.
Adding symbols and footprints to user libraries does not require refreshing the table files.
Project Personal libraries are only available to access when the particular project they are related to is open.
Global Personal Libraries are available all the time.
Default libraries bundled with KiCad and updated by each version change.
3d party libraries, installed by plugins. These are added in fp-lib-table and sym-lib-table at KiCad user configuration directory by plugin.
This is exactly my problem that I described at begin. KiCad doesn’t do that automatically nor give some semi-automated tools to migrate these between users and systems. Without editing fp-lib-table and sym-lib-table manually copied library subdirectories doesn’t appear as new groups in symbol and footprint tree.
Sigh… Seems I should untie my Python proficiency and make a new plugin that do exactly what I need.
How would KiCad know which libraries you want to add? You have to tell it yourself. Now for doing it for many accounts, maybe you could find a way to reduce the work.
Yes that’s always been the case, just populating the directories doesn’t make them appear in the browser/chooser. The GUI actions Manage Symbol/Footprint Libraries do exactly that, modify those table files.
BTW a common misconception is that a path variable is a search path. It is not. It’s simply a string substitution. The predefined variables (which can actually be overriden) are preset to where the packages have installed the standard libraries. Use of these predefined variables is what allows KiCad to use the same template tables across OSes.
As I said, progress on this front depends on the devs finding a way to decouple system library entries and user library entries so that the latter can be modified easily without affecting the system library entries.
You can make a (backup) copy of the complete KiCad configuration directory on your work PC, and then compare it with your setup at home with a sourcecode merge program such as meldmerge. This is also a good help to integrate changes you made to your KiCad setup into a new mayor version of KiCad. But do be aware that different mayor KiCad versions may have slightly different syntax in their configuration files.
Editing the configuration directory this way is relatively safe, especially if you make a backup copy of that whole directory. Even if you *&^%$#@! and damaged files in that directory, you can simply delete the whole directory. As a result KiCad creates a new configuration directory from defaults the next time it is started. As a side effect, this of course results in you loosing all personal settings you made, so working with a backup by simply copying the whole directory under a different name is recommended.
you can use file system link and you add a link in the kicad config folder to point to the root of the library folder and you have there *-lib-table file, so that the file is synced. But then each user has the same *-lib-table file.
If you use git for library syncing there might be a way to add a post-merge hook script which would trigger sync between personal *-lib-table file and the one in the library folder
Yes, I did that today as temporary measure. It has a drawback though - all 3d party library subdirectories added to somebody local KiCad configuration by plugins will appear in other people configurations. But in short term it will help.
I don’t exactly understand your problems as it looks they are much wider than the way I use libraries.
For me very helpful is that I use only my own libraries. So when upgrading to new version at first run of Schematic or PCB I just select from given me choices that I want to use the same list as in previous version. KiCad asks me to select right library list file, I select it and done - I’m in new version.
The only problem is that before this first run I have to temporarily copy my library list to some directory that is not system/hidden. Library lists I have done with KiCad V4 are all the time with me.
I have in my daily backup my KiCad directories. When I want to work at home I don’t have to do any preparations to this - I have everything in my backup copy I have always with me.
I just wanted suggest you that may be problem you try to solve will be simplified if you decide to not use KiCad libraries. If I were using my and KiCad library I will have to edit library lists at each version change. As I use only my libraries I don’t have to do that.