I think that can be probably solved for the most part by simply rebranding them as “substitution variables” that can be optionally seeded from environment variables when needed but override them. Which is actually accurate description of what they are and how they work currently.
The fact that they are processed through wx is an implementation detail, you are just using wx env store as a global hashmap, you can just as easily add a static var to PGM or another global.
Either way, the above is tangential to my idea that I’ll try to clarify:
When saving a file pcbnew/eeschema passes a list of all substitution variables (just names without values) to project file. There already is some logic to do that for board design and erc settings so the channel to piggy back on already exists.
When parsing project file check that all substitution variables are known (either from config or from actual environment variables, doesn’t matter) and prompt to define them if necessary or mark as ignored.
Nothing here changes how the variable evaluation or definition is implemented.
Maybe, but there’s not an easy mechanism right now to build a list of all variables in use. This seems like a big chunk of work for small gain to me, but open an issue for it if you think it’s imporant and we can see how many votes it gets.
One enhancement that could be useful is a command line switch to use a different config directory. It would also allow a user to observe another user’s problems.
Maybe there is some overlap that I’m not seeing, but it sounds like “environment variable” is used to mean two separate concepts here. In the link it appears KICAD_CONFIG_HOME is what I’d call an environment variable in the traditional sense (i.e. something system-wide and not necessarily tied to KiCAD and something you can echo from any terminal) The “environment variables” stored in kicad_common.json defining the path substitutions are not something you could echo from the terminal (i.e. they are in a specific config file for KiCAD use only). Is there any reason the latter ones can’t also be traditional environment variables? In this way one could just source a config file that contains all these per project. So instead of
As it stands, since it is in a json file, I’d have to make use of sed or some other text hacking to change these via a script, and that is inherently not very forward compatible. Maybe I just am not aware of this functionality already existing? Does KiCAD check for other traditional environment variables other than KICAD_CONFIG_HOME? I.e. can I set KICAD6_3DMODEL_DIR in my terminal and it will override the one in kicad_common.json?
All the other things in kicad_common.json make sense to me, things that really only affect the GUI itself, and are project independent. If I need to work on a project with someone else, they don’t have to share all my GUI preferences, but they have to share my paths. These paths in the “environment.vars” section seem like they should be stored elsewhere, and I’d vote for them to be stored in a file that would act in a similar way to how I source any other environment variables from a terminal.
This isn’t really true. It is quite possible to collaborate on projects without having the same paths configuration shared between machines.
You either can to manually configure the same variables to point to appropriate places (note they don’t need to point to the same places) or you can use project-local libraries that don’t use variables at all. Neither of these approaches requires sharing the configuration inside kicad_common.json. @qu1ck was proposing one possible mechanism for making the first approach more straightforward, but even without any code changes, I personally find it very easy to do one-time setup on each machine if using the variables approach.
Note that you should not be changing KICAD6_*_DIR! These are for the built-in KiCad libraries, not for your custom project-specific libraries.
Yes, you are right, sorry I spoke to generally. Yes, I meant more specifically that we must share the same contents of some local path (e.g. via git and github to point to our own respective local repos).
I’m all for the detection proposed by @qu1ck, but thats not quite the issue I’m talking about. I’m talking about working on multiple projects in parallel, and having to switch between them quickly throughout the day. If I only had to set these variables once in a few months, I agree, that your solution would be somewhat acceptable. Because of how often I need to switch between projects, I’d ideally like to have some mechanism similar to what I use in C or Python development where I either use temporary environmental variables via what I described in my last post, or something like virtual environments that I’d use in python to store all these kinds of configurations specific to a given project. Both of those mechanisms and what I proposed in my last post would be a single line in the terminal
source some_config_file
to switch between project-specific configurations.
Sorry that was not very well chosen as an example without context. You’re right, I should have chosen less core values in my example. That being said, it brings up an interesting tangential point. I do in fact change these values intentionally because I manage a fork of the built-in KiCad libraries. This allows me to:
Make PRs for new symbols/footprints/3dmodels to add to the built-in KiCAD Libraries
Pull any new symbolts/footprints/3dmodels
Checkout different branches that correspond to different projects. (i.e. I can have a branch that is behind the current main branch to prevent any new updates from breaking my existing projects).
Yes, that is one option, but it leads to a lot of duplicate parts in my experience. Lets say I have a custom part (symbol+footprint+3d model). I want to use this in multiple projects. I could either duplicate the part in both local libraries, which would lead to maintenance issues and bulk, or I can maintain this in my fork of the built-in KiCAD Libraries. You might then want to know how I can share this without making someone download a huge amount of data duplicating the built-in libraries. Its easy with git! So now when I want to share my custom parts with people, they can just track my fork’s project-specific branch using
git remote add topherbuckley <someURL or SSH>
then they don’t have to download or duplicate the entire contents of the built-in KiCAD libraries, they only track the changes, and pull in the changes.
Can you explain more why you are switching variables when you switch projects? This part makes no sense to me and isn’t part of any workflow I’ve seen before.
This is OK if you understand what you are doing, but it’s an advanced use case that most users shouldn’t be pointed to. I would personally handle this by leaving the read-only copy of the libraries alone, and adding a second editable copy of any library that I’m working on.
There is a third option, which is the one I see people using the most anecdotally: put your custom parts into a custom library that lives in a standalone Git repository, then reference this custom library in the project library tables (not the global ones). You can make this reference either by relative paths or by env vars, but either way, you aren’t actually storing the library contents once per project, you’re just storing a link to a library that you then share.
It seems to me like you are trying to “fight against the system” rather than work with it, by creating a fork of the entire KiCad libraries and then putting project-specific changes into branches in that fork. I think you might have a much easier time with this if you left the official libraries alone, and put your changes/additions into a different Git repository that is not related to the KiCad upstream libraries.
As long as someone is comfortable with git, then I’d definitely point them to this method as there are a bunch of advantages (revision tracking, deduplication, easy swapping, etc.), but sure if someone isn’t using git, then local copies etc can work (though I cringe at the inefficiency and instability). I don’t recall how this worked in the past, but if I’m installing kicad via apt-get in ubuntu, It automatically downloads all these core libraries (sym/lib+3d) again and I’ve had instances in the past where my changes to any built ins were completely wiped due to an update. Yeah, I should have made a copy like you say to somewhere else, but I didn’t know an update would replace my files until it happened. (Thus now I use git and have revisions and backups among all the other advantages I mentioned).
Yep, I actually use something like this as well. It isn’t a mutually exclusive solution. I use it in tandem with the other things I mention here. It has its use case, just like the other options.
Sounds like you may have misunderstood me. I’m not developing against the core libraries, I’m developing with them. I submit PRs to them from time to time (though very infrequently), and I’m always pulling in updates to maintain a relation with the most recent changes. For example, I also have a few custom libraries that will never be merged into the core ones (as they don’t meet the standards required or some other reason) but I still keep them within the same repo (i.e. I have one repo for symbols, one repo for footprints, and one for 3d models, just like they are maintained by KiCAD) I just maintain a fork so I can add to it. Its a pretty standard use case of git.
Sure, glad to!
For the sake of this example, lets say I’m working on three different projects (P1, P2, and P3). To start, lets say there is only one other person working on each project other than myself (things get even more complicated when there are more than 1). Lets say P1 has a default kicad setup (i.e. all paths in their kicad_common.json are the default installed one) and they never modify the core libraries. Lets say the P2 developer manages their own copy of the core libraries like you mentioned you do so they can customize some things. They refer to this as “P1_KICAD7_SYMBOL_DIR”. Finally the P3 developer has modified several symbols in the core libraries and has done it in place (i.e. no other copy, they are modifying things that will get overwritten upon updating. No, P3 should not be doing this, and it will cause them all kinds of headaches, but they didn’t know this, and their schematics are already done and ready for me to review. Realistically I see a LOT of P3 type projects. Since you can’t really control how clients will do this, and you still have to work with them, you have to deal with it somehow.
For P1 it might be easy, and it might not. We might be on different versions of kicad or the libraries might have been updated via apt-get and I updated but they didn’t. Theres all kinds of things that could potentially go wrong and most of the time silently. In an ideal case, I already have the same libraries as they do; nothing to import nothing to change other than I checkout the core lib branch on git that matches theirs. All of these things are no longer ambiguous but rigidly defined if you have matching configurations to relative paths (its the same way dependencies in CMake or Python’s virtualenv work. Its a tried and true way of doing it.
P2 has a modified copy of the core sym+fpt+3d libraries that I need to download. I can download a copy of their custom libraries, but now I have 2x roughly 6 GB in library files even though their project only really uses 1% of them. This both a waste of space and doesn’t scale well, but it works. I can add each of their libraries to my kicad_common.json file as some separate library “P1_KICAD7_SYMBOL_DIR” and work on their projects. This seems to be your workflow from what I understood? Its not a deal breaker, but this also appears to mean you have to maintain an ever growing list of these “environmental variables” adding some for each project. For me, I’d like to keep things concise and only be loading in libraries relevant to the current project (and the core ones). If I’m working on P1 I don’t want P2 libraries cluterring my search results. There are going to be a lot of duplicates due to this as well (e.g. say I search for 1N4001 diode. There will now be two versions with the same name.). For this example its only one library, but some projects have quite a few custom libraries and this doesn’t scale well. Reading through the entire list and clicking a toggle switch for each is much more time consuming and error-prone than a sourceable config file specific to a project.
The real problems come in when working with P3 type projects. Here their project references “KICAD7_SYMBOL_DIR”, which are different from the libraries defined in the core “KICAD7_SYMBOL_DIR”. Even if I added their libaries to some other path and used another “environmental variable” name, all the symbols in the P3 project will resolve to the core version of the symbols. I’ve hacked around this a few times, but its all very error prone unless someone has alternative solution for this case? Worse still, try to scale that to another 10 clients who all made their designs like P3.
One other good example is related to what you referred to as “a third option”. Say I have some custom library, and another person has to make use of it to open/edit my project. Lets call it MY_CUSTOM_LIB. Yes, it would be easy enough for this person to add this variable to their Configure Paths dialog if it were a one time thing, but think about scaling that. What if I have 300 custom libraries? This is no longer a viable option. You might ask why I would need so many, and its because I like to be organized. I don’t like having all my custom parts in a single library for the same reasons KiCAD doesn’t put all the built-in parts into a single library. There is a custom diode library, a custom mosfet library, etc. Again this plays into the reasoning for P2, in that I don’t want some P2 client’s 300 custom libraries all showing up in my Configure Paths dialog and having to unclick 300 things every time I want to look at a different project.
@eelik do you know if there is a reference somewhere listing all the true system environment variables that kicad looks for beside KICAD_CONFIG_HOME? I’m curious to know what I can all play with here.
For dealing with clients, in some cases it seems like you may not need libraries at all (after all, symbols/footprints are stored as copies in the design files).
But still, for all these cases, I don’t think the right answer is for you to continually reconfigure KiCad’s built-in libraries (by changing environment variables around). The right answer in my opinion is to convert these projects to use project-specific libraries if you need to edit the libraries, or simply ignore the issue if you don’t need to edit the libraries.
It seems like you have a system worked out that works for you, but you also started this thread because you are running into friction. My message is just that I think you are running into friction because you’re trying to work in a way that is not really supported or advised.
The difference is, I don’t refer to these libraries in projects and I don’t set up environment variables pointing to them.
Why do you need to download them? Either you don’t need to modify the library symbols, or you can modify them in-place, or you can export the symbols/footprints from the design files into a new library and modify that. You don’t need to have a complete copy of someone else’s libraries in order to work on designs that they made.
Wasn’t that a v6 change? I recall this not always being the case. Do they also store 3d models?
If every client I had was just a one-off project, I’d agree with you, but I don’t want to fix their libraries with every project I do for them, so I’d rather I just fixed them for them, so next time they come to me they have the updated ones. Git makes this super easy, but not everyone uses it. If they don’t then I just download the whole thing and send it back to them.
I realize that, but what I propose with a project-specific config file that can be sourced is the way its done in any other development framework I’ve used. I can only speak for the platforms I develop on, but the problems we’re discussing here are (some of) the reasons why C developers use CMake, python developers use virtualenv, and Android devs use Gradle. This is a major motivator for Docker and Singularity as well. Isolating dependencies to a specific project rather than having everything system-wide is a very important part of repeatable development. Having the ability to treat KiCAD development with the same level of repeatability/stability is a HUGE advantage over the industry standard alternatives (Altium, etc.). The text based source files (as opposed to binary like Altium) in KiCAD and the ability to use it with git is one of the major reasons I use it.
Yes, it was a V6 change, but we’re now halfway through V7 so not a super recent change at this point. No, 3D models are not included yet, but 3D models can be referenced through relative paths at least. In the future it will be possible to store 3D models inside boards, so that no libraries will be required for portable 3D model viewing/exporting.
KiCad already supports project-specific library configs, though. Using these, you can have an isolated, project-specific configuration, and if these project-specific library tables are pointing at footprint libraries that use relative-pathed 3D models, you also can have portable 3D models without requiring environment variables to be set up. The only thing it doesn’t support is project-specific overrides of the global libraries, which is what you see to be looking for.
If we are talking about making edits to the KiCad official libraries, I just open the read/write copies in the library editor.
For repeat clients, why not take the time to set them up with a custom library setup that only includes any library parts that are actually customized for them, rather than a library based on the KiCad standard library that ends up being 6GB?
No, but Murphy’s Law still tells me I’ll still have a few projects that use < v6 :D. I started this workflow sometime back in v4 from what I recall, so I’ll keep my backward compatibility for now.
Good to know! I think this is all great, for its own reasons, but for the reasons I mentioned before I don’t like the idea that I’m working on a copy of someone’s work rather than contributing to bettering their libraries. This kind of mindset is what really turned me away from Altium and the like where you’re exporting projects to share, and hundreds of duplicate copies are floating all over the place.
I think I’m confused. Can you walk me through when you’d do this and for what? You’re not modifying the official libraries? If not, and you have another copy somewhere, don’t you have to reference it somehow via a environmental variable defined in kicad_common.json?
Because they don’t pay me to fix ALL their customized parts, nor do I have the time. If a project I’m working on uses say 50 parts, and only 3 need revisions, its a big stretch to ask me to review every customized part when I can just fix three, contribute to their revisions, and keep moving. They might have hundreds or thousands of customized parts. Its the difference between me understanding the entire KiCAD source vs adding a new part to a existing library. Its quite a bit difference, and I definitely am nowhere close to understanding all of the former
Say I wanted to modify something in the official library Connector. I’d make a clone of the git repository, start a new branch, and then add the Connector library with a different nickname to my global library table. I wouldn’t bother setting up an environment variable prefix for the path to this library, since I only need this on one machine, so the full path to the copy of the library would sit in my global library table. No variables involved. I would not place this modified symbol into a design; I’d only use it for proposing a change to the official library. If I wanted to place it into a design, I’d first copy it into a project-specific symbol library, or into a private shared library (i.e. one that is used across many projects, but is not based on the upstream KiCad official library repo)
Anyway, for your particular workflow, I have no further suggestions if you don’t want to change your workflow. If you want to be editing the client’s libraries, rather than editing a copy of them, and that client has improperly configured KiCad so that their working (editable) libraries are the global KiCad stock libraries, there is no good way to handle that other than advising the client that they should fix their setup.
In an ideal world, yes this would be the perfect workflow and everyone of my clients would use git and just fetch the updates , and I wish I could take the time to meet the standards for every custom part I make, but I rarely have time for this (yes a weak excuse). Also PRs can take months/years sometimes, so waiting to use it isn’t always an option.
To each their own. If that works well for you, then great. It too closely resembles what I didn’t like about the Altium way of doing things though. So what happens when your changes get merged into the official library repo? Do you keep track of them all and go back and reference the official repo versions or keep using the ones in your project-specific library or private shared library? You now have two diverging copies to maintain, and that divergence only grows with each project. Sounds tedious and messy to me, but maybe you have a good system of dealing with all this.
Since I haven’t tried to edit them in quite some time, I’m curious what are the write permissions on the core libraries these days? Are they installed with read-only by default? Was this always the case? Maybe I’m just dealing with something that less and less people will do on accident nowadays?
I use to work with multiple computers/OS with the same project. I never have to configure an environment variable for Kicad. I also work with others and also using git.
The core (read-only) library is the default for everyone. They should just have the initial library setup using the Default Kicad Libraries.
After that, I also use custom libraries per project.
Here, the only rule I have is that this library has to be inside the project, so the project will be always pre-configured to find those libraries. This library folder inside of the project can be the library itself, but it can also be a copy of the library folder, a git submodule, or a symlink to the real path library path. It doesn’t matter which one it is, this will always work. Also, the project has to have the fp-lib-table and the sym-lib-table with references to the libs that are inside the project.
This works everywhere and also on every OS including Windows.
If the real lib is outside the project, you have to share 2 things (the project and the lib) if needed (since the lib is not always needed once Kicad files are enough, except for the 3d models, I guess) and then you have to recreate the symlink if it was not created relative to the project path, and also if the used lib path doesn’t exist.
So now when I want to share my custom parts with people, they can just track my fork’s project-specific branch using