Is there anything new in 7.0 that would lead to easier portability/sharability? Specifically I mean I want to be able to work with someone else on the same project across machines. Usually I’d use git+GitHub for this. For the actual project files, kicad_sch, etc. files git works great. The big issues come due to the dependencies on your sym-lib-table, fp-lib-table and kicad_common.json config files. Without sharing these files, and putting them to use, the project’s symbols and footprints won’t be found. Collaborators must be using these same versions of these files to work together (unless I’ve missed some other new development).
You can store these files in the git repo for your project, and then use symlinks to put them back in the expected directories (e.g. ~/.config/kicad/7.0 on Ubuntu), but this makes swapping between projects pretty cumbersome and likely error prone for those not keeping backups or git versioning of these files. (i.e. if someone replaces their existing three files with someone else’s without a backup, I can see this causing a lot of stress on someone).
Is there no way of temporarily changing these three files to point towards someone else’s project ones within KiCAD? Ideally I would think this should be just another config value you can change from within KiCAD, (e.g. KICAD_CONFIG_DIR). For example, I know that when you first open KiCAD after installing some update it will ask if you want to use the default versions in the installed directories for your sym-lib-table, and fp-lib-table files, but after you’ve specified these, how can you point to some other version? For example, lets say I select some custom version when I open KiCAD for the first time, then I decide I want to go back to the default tables. Is there a way to do that without reinstalling?
Reinstalling isn’t even the cure for returning to the dialogue for choosing the library tables as they persist across reinstalls but moving or deleting the two tables will work.
You are not intended to share kicad_common.json (or other config files) between users – it is not necessary for sharing project files between machines.
The symbol and footprint library tables in the config folder are also “machine-specific” global configs. It sounds like you may want to use project-specific library tables, which are designed to be stored next to your project files.
Without the contents of kicad_common.json, the Environment Variables (Preferences->Configure Paths) defining the path replacements will not match the values defined throughout the project so again all symbols and footprints will not resolve. Do you know a way around this?
Also it is a fair pain to change every path (I’ve got 10-15 of them in use) for every project via a file open dialog every time I want to change between projects.
You can either have all users define the same environment variables (manually, not by sharing files) or you can not use environment variables and instead just use relative paths.
It would be cool if upon opening a project for the first time if kicad encounters unknown environment variable it would prompt user to define it (with an option to ignore which would lead to current behavior).
I think the problem there is KiCad doesn’t nessacarily evaluate variables just when you open a project. You have to trigger either displaying it/using it. Changing that would be a rather large change and difficult to maintain.
You could approach it from another angle: cache used env vars in the project file so that there is no need to evaluate board/schematic files. Pcb and schematic editors would still reevaluate them as needed so no change in their logic or file formats.
Would that still be a big change?
Yes. Environment variables are processed through wx, not through the project.
We’d be better off encouraging people not to use environment variables and introducing other mechanisms where needed to allow people to share library configs.
I think that can be probably solved for the most part by simply rebranding them as “substitution variables” that can be optionally seeded from environment variables when needed but override them. Which is actually accurate description of what they are and how they work currently.
The fact that they are processed through wx is an implementation detail, you are just using wx env store as a global hashmap, you can just as easily add a static var to PGM or another global.
Either way, the above is tangential to my idea that I’ll try to clarify:
When saving a file pcbnew/eeschema passes a list of all substitution variables (just names without values) to project file. There already is some logic to do that for board design and erc settings so the channel to piggy back on already exists.
When parsing project file check that all substitution variables are known (either from config or from actual environment variables, doesn’t matter) and prompt to define them if necessary or mark as ignored.
Nothing here changes how the variable evaluation or definition is implemented.
Maybe, but there’s not an easy mechanism right now to build a list of all variables in use. This seems like a big chunk of work for small gain to me, but open an issue for it if you think it’s imporant and we can see how many votes it gets.
One enhancement that could be useful is a command line switch to use a different config directory. It would also allow a user to observe another user’s problems.
Maybe there is some overlap that I’m not seeing, but it sounds like “environment variable” is used to mean two separate concepts here. In the link it appears KICAD_CONFIG_HOME is what I’d call an environment variable in the traditional sense (i.e. something system-wide and not necessarily tied to KiCAD and something you can echo from any terminal) The “environment variables” stored in kicad_common.json defining the path substitutions are not something you could echo from the terminal (i.e. they are in a specific config file for KiCAD use only). Is there any reason the latter ones can’t also be traditional environment variables? In this way one could just source a config file that contains all these per project. So instead of
As it stands, since it is in a json file, I’d have to make use of sed or some other text hacking to change these via a script, and that is inherently not very forward compatible. Maybe I just am not aware of this functionality already existing? Does KiCAD check for other traditional environment variables other than KICAD_CONFIG_HOME? I.e. can I set KICAD6_3DMODEL_DIR in my terminal and it will override the one in kicad_common.json?
All the other things in kicad_common.json make sense to me, things that really only affect the GUI itself, and are project independent. If I need to work on a project with someone else, they don’t have to share all my GUI preferences, but they have to share my paths. These paths in the “environment.vars” section seem like they should be stored elsewhere, and I’d vote for them to be stored in a file that would act in a similar way to how I source any other environment variables from a terminal.
This isn’t really true. It is quite possible to collaborate on projects without having the same paths configuration shared between machines.
You either can to manually configure the same variables to point to appropriate places (note they don’t need to point to the same places) or you can use project-local libraries that don’t use variables at all. Neither of these approaches requires sharing the configuration inside kicad_common.json. @qu1ck was proposing one possible mechanism for making the first approach more straightforward, but even without any code changes, I personally find it very easy to do one-time setup on each machine if using the variables approach.
Note that you should not be changing KICAD6_*_DIR! These are for the built-in KiCad libraries, not for your custom project-specific libraries.
Yes, you are right, sorry I spoke to generally. Yes, I meant more specifically that we must share the same contents of some local path (e.g. via git and github to point to our own respective local repos).
I’m all for the detection proposed by @qu1ck, but thats not quite the issue I’m talking about. I’m talking about working on multiple projects in parallel, and having to switch between them quickly throughout the day. If I only had to set these variables once in a few months, I agree, that your solution would be somewhat acceptable. Because of how often I need to switch between projects, I’d ideally like to have some mechanism similar to what I use in C or Python development where I either use temporary environmental variables via what I described in my last post, or something like virtual environments that I’d use in python to store all these kinds of configurations specific to a given project. Both of those mechanisms and what I proposed in my last post would be a single line in the terminal
source some_config_file
to switch between project-specific configurations.
Sorry that was not very well chosen as an example without context. You’re right, I should have chosen less core values in my example. That being said, it brings up an interesting tangential point. I do in fact change these values intentionally because I manage a fork of the built-in KiCad libraries. This allows me to:
Make PRs for new symbols/footprints/3dmodels to add to the built-in KiCAD Libraries
Pull any new symbolts/footprints/3dmodels
Checkout different branches that correspond to different projects. (i.e. I can have a branch that is behind the current main branch to prevent any new updates from breaking my existing projects).
Yes, that is one option, but it leads to a lot of duplicate parts in my experience. Lets say I have a custom part (symbol+footprint+3d model). I want to use this in multiple projects. I could either duplicate the part in both local libraries, which would lead to maintenance issues and bulk, or I can maintain this in my fork of the built-in KiCAD Libraries. You might then want to know how I can share this without making someone download a huge amount of data duplicating the built-in libraries. Its easy with git! So now when I want to share my custom parts with people, they can just track my fork’s project-specific branch using
git remote add topherbuckley <someURL or SSH>
then they don’t have to download or duplicate the entire contents of the built-in KiCAD libraries, they only track the changes, and pull in the changes.
Can you explain more why you are switching variables when you switch projects? This part makes no sense to me and isn’t part of any workflow I’ve seen before.
This is OK if you understand what you are doing, but it’s an advanced use case that most users shouldn’t be pointed to. I would personally handle this by leaving the read-only copy of the libraries alone, and adding a second editable copy of any library that I’m working on.
There is a third option, which is the one I see people using the most anecdotally: put your custom parts into a custom library that lives in a standalone Git repository, then reference this custom library in the project library tables (not the global ones). You can make this reference either by relative paths or by env vars, but either way, you aren’t actually storing the library contents once per project, you’re just storing a link to a library that you then share.
It seems to me like you are trying to “fight against the system” rather than work with it, by creating a fork of the entire KiCad libraries and then putting project-specific changes into branches in that fork. I think you might have a much easier time with this if you left the official libraries alone, and put your changes/additions into a different Git repository that is not related to the KiCad upstream libraries.
As long as someone is comfortable with git, then I’d definitely point them to this method as there are a bunch of advantages (revision tracking, deduplication, easy swapping, etc.), but sure if someone isn’t using git, then local copies etc can work (though I cringe at the inefficiency and instability). I don’t recall how this worked in the past, but if I’m installing kicad via apt-get in ubuntu, It automatically downloads all these core libraries (sym/lib+3d) again and I’ve had instances in the past where my changes to any built ins were completely wiped due to an update. Yeah, I should have made a copy like you say to somewhere else, but I didn’t know an update would replace my files until it happened. (Thus now I use git and have revisions and backups among all the other advantages I mentioned).
Yep, I actually use something like this as well. It isn’t a mutually exclusive solution. I use it in tandem with the other things I mention here. It has its use case, just like the other options.
Sounds like you may have misunderstood me. I’m not developing against the core libraries, I’m developing with them. I submit PRs to them from time to time (though very infrequently), and I’m always pulling in updates to maintain a relation with the most recent changes. For example, I also have a few custom libraries that will never be merged into the core ones (as they don’t meet the standards required or some other reason) but I still keep them within the same repo (i.e. I have one repo for symbols, one repo for footprints, and one for 3d models, just like they are maintained by KiCAD) I just maintain a fork so I can add to it. Its a pretty standard use case of git.
Sure, glad to!
For the sake of this example, lets say I’m working on three different projects (P1, P2, and P3). To start, lets say there is only one other person working on each project other than myself (things get even more complicated when there are more than 1). Lets say P1 has a default kicad setup (i.e. all paths in their kicad_common.json are the default installed one) and they never modify the core libraries. Lets say the P2 developer manages their own copy of the core libraries like you mentioned you do so they can customize some things. They refer to this as “P1_KICAD7_SYMBOL_DIR”. Finally the P3 developer has modified several symbols in the core libraries and has done it in place (i.e. no other copy, they are modifying things that will get overwritten upon updating. No, P3 should not be doing this, and it will cause them all kinds of headaches, but they didn’t know this, and their schematics are already done and ready for me to review. Realistically I see a LOT of P3 type projects. Since you can’t really control how clients will do this, and you still have to work with them, you have to deal with it somehow.
For P1 it might be easy, and it might not. We might be on different versions of kicad or the libraries might have been updated via apt-get and I updated but they didn’t. Theres all kinds of things that could potentially go wrong and most of the time silently. In an ideal case, I already have the same libraries as they do; nothing to import nothing to change other than I checkout the core lib branch on git that matches theirs. All of these things are no longer ambiguous but rigidly defined if you have matching configurations to relative paths (its the same way dependencies in CMake or Python’s virtualenv work. Its a tried and true way of doing it.
P2 has a modified copy of the core sym+fpt+3d libraries that I need to download. I can download a copy of their custom libraries, but now I have 2x roughly 6 GB in library files even though their project only really uses 1% of them. This both a waste of space and doesn’t scale well, but it works. I can add each of their libraries to my kicad_common.json file as some separate library “P1_KICAD7_SYMBOL_DIR” and work on their projects. This seems to be your workflow from what I understood? Its not a deal breaker, but this also appears to mean you have to maintain an ever growing list of these “environmental variables” adding some for each project. For me, I’d like to keep things concise and only be loading in libraries relevant to the current project (and the core ones). If I’m working on P1 I don’t want P2 libraries cluterring my search results. There are going to be a lot of duplicates due to this as well (e.g. say I search for 1N4001 diode. There will now be two versions with the same name.). For this example its only one library, but some projects have quite a few custom libraries and this doesn’t scale well. Reading through the entire list and clicking a toggle switch for each is much more time consuming and error-prone than a sourceable config file specific to a project.
The real problems come in when working with P3 type projects. Here their project references “KICAD7_SYMBOL_DIR”, which are different from the libraries defined in the core “KICAD7_SYMBOL_DIR”. Even if I added their libaries to some other path and used another “environmental variable” name, all the symbols in the P3 project will resolve to the core version of the symbols. I’ve hacked around this a few times, but its all very error prone unless someone has alternative solution for this case? Worse still, try to scale that to another 10 clients who all made their designs like P3.
One other good example is related to what you referred to as “a third option”. Say I have some custom library, and another person has to make use of it to open/edit my project. Lets call it MY_CUSTOM_LIB. Yes, it would be easy enough for this person to add this variable to their Configure Paths dialog if it were a one time thing, but think about scaling that. What if I have 300 custom libraries? This is no longer a viable option. You might ask why I would need so many, and its because I like to be organized. I don’t like having all my custom parts in a single library for the same reasons KiCAD doesn’t put all the built-in parts into a single library. There is a custom diode library, a custom mosfet library, etc. Again this plays into the reasoning for P2, in that I don’t want some P2 client’s 300 custom libraries all showing up in my Configure Paths dialog and having to unclick 300 things every time I want to look at a different project.
@eelik do you know if there is a reference somewhere listing all the true system environment variables that kicad looks for beside KICAD_CONFIG_HOME? I’m curious to know what I can all play with here.
So now when I want to share my custom parts with people, they can just track my fork’s project-specific branch using