I’m planning to work with several other people in parallel on a bunch of PCBs using Gitlab. I wntad to know if you people have done this in the past and what your experiences/lessons learned are.
My main concerns are:
using common libraries
how do we prevent users from mixing our libs with their local ones?
what are common problems when having multiple users using the same git-based libraries
as I understood, the KIGITHUB option is pretty crippled and not recommended
Git with KiCad is very good for versioning but not good for distibuted development.
The main rule: distributed development within one file isn’t possible. That’s the nature of the data, which is basically 2D graphics where location on the canvas doesn’t correspond with the location in the file. When conflicts arise they are very difficult or impossible to solve.
Subsheets might work for schematic. There’s no solution for the board file.
Very annoyingly KiCad stores non-relevant data to files, e.g. layer visibility in the board file. Even with sequential editing such changes go between users.
In the same way than you would prevent it without git, I think. The best solution might be to have self-contained project which has project specific libraries. The needed components would be copied to them from the global common libraries. Global libraries wouldn’t be used at all directly.
Footprints on the board are copies anyways and aren’t updated automatically even when the library is changed. Symbols may cause more problems with global libraries; add sym-lib-cache to the project specific sym-lib-table.
It’s a really interesting question to me and good advice from @eelik. We are using Git as the back-end for people to upload electronics projects to kitspace.org but this is mainly for the file-storing and versioning aspects. Merging hasn’t really been made possible by any tools yet.
I’d be super interested to hear how your project goes and what you learn from it though. Please do write it up somewhere.
This is not a rare situation by any means. It’s a topic I’d like to see addressed at gatherings such as “KiCon”. I’d like to know how others have dealt with the problem - whether their solution is a good one, only mediocre, or still problematical, I suspect everybody would benefit from hearing and discussing the ideas.
The ancient fable about money and ideas applies:
If I have a dollar, and you have a dollar, and we exchange dollars, then we simply preserve the status quo and neither of us gains anything. But if I have an idea, and you have an idea, and we exchange ideas, then we are both enriched by having two ideas.
We work with github and KiCad with several projects.
Libraries:
We keep our libraries in a folder into the project kicad folder (the folder with the project.pro, schematic and layout files, etc.), all with relative paths from $KIPRJMOD.
The folder libraries contains:
-Our general library with 3 subfolders:
Symbols -> general.lib general.dcm
Modules3D
general.pretty
-Our project specific library with 3 subfolders too:
Symbols -> project.lib project.dcm
Modules3D
project.pretty
Every project has a full copy of the general library. Saves space only in the repo, not in the local drive.
The general library is set as a submodule, meaning that must be pushed independently of the rest of the project.
After every update of the general library, it must be pulled by all the projects. I made a sript for this task.
In the past years, the general library didn’t exist. I just copied the needed components into the project library from a general library kept in my hard disk for each project and the libraries were updated within their projects.
The project folder is updated within each project.
Another option is to keep the libraries outside the project folder. Then there is no need of a submodule, just use the libraries as an idependent project.
Only one person must work in a specific library at a time.
Schematic:
Working with subsheets should be a must so each developer makes commits of his/her own work not touching the other sheets.
Layout:
It is difficult to work in parallel, not to say impossible. We have stated a check-in check-out flag via an issue in github to tell the other one if we are working or not in the project. Otherwise there are merging conflicts.
We are only two people working in the common projects. The main problem raises when someone (always me) forgets to push changes to the repo.
There are a bunch of talks about revision control and git planned at the conference. I could also insert a “workflow” panel if you think that would be an interesting segment/panel to hold with multiple people on stage.
Form experience i can say that merging library assets is possible. (But requires experience to get it right)
I would however refrain from doing this on a schematic or pcb file. Leaving you with the option of either iron discipline or a version control system that allows to lock files. (plus a workflow that guarantees that users get their locks reliably before opening the files. This might need some sort of script or again iron discipline.) This means you will need to use a centralized version control system like svn. (Or shop around. There are such tools out there that guarantee that only you can edit a file.)
A general remark on library organization. I am a fan of centralized libs with a well defined set of rules and harsh punishments for not following them. (A whip might do the trick Or a round of everyones favourite beverage at the next organization wide evening activity. Coffee or pizza could also do the trick.)
To achieve this have a centralized storage place (example a special directory in the version control tool or maybe even a separate repo.) and somebody responsible for at least training everyone on the set of rules. (Ideally only a set few persons can write to the central lib. But that is really only viable in larger organizations.)
Make sure you include and maintain library tables. And use personalized path variables. (Especially for 3d models. Or find out how the new 3d model searchpath feature works.)
Sadly your users will need to update their library table from time to time, possibly at the start of each new project depending on how often you add new libs. (Man i wish kicad would either auto parse a given directory, provide a way to have multiple lib tables or at least a lib table import feature)
I would then also have project local libs for these parts that are really specialized to that project (The symbol of the MCU that encodes how its pins are configured. Connectors with well defined pin configs. Specialized power symbols. …)
My main reason against having mainly project local libs is that you will end up with the same assets in many projects with everyone in a different state/standard. You can not really say i used this a lot of times and know it works if you do not really know which version of the asset worked last time. (And how would you know who to whip if you do not know who does not follow the rules. Or do you want to go through everyones local files just to find that out?)
But footprints are copied to board file anyways, so if they must be kept up to date, one must run Update Footprint for the board every time a change is made in the global library. And forcing footprint changes to existing designs may mean lots of other changes especially if the design is tight.
So, I would say that first it must be decided whether keeping board footprints up to date is wanted or not.
This was more along the lines that if you have assets local to projects then there is a high likelihood that somebody else in the team will not find that asset if they also need it. So they will redesign it. (Now what happens if one of these has a mistake. Some third party might remember that there was a successful board produced with such an asset but might not remember which project that was. They now have a 50/50 chance.)
I was thinking about a “waterfall” where changes to assets (footprints) are made upstream, centralized. The latest versions would be copied to project when they are used there.
I don’t think there’s one best alternative. Think about the situation where a component is added to the board after the board has already been partially done. There are, say 0402 R footprints in the board. The project uses global library. The footprint has been changed after the project started, so the old parts of the design have the old version of the footprint. (This happens of course regardless of globality or locality of the libary because footprints become part of the board and aren’t updated automatically.) Which version of the footprint the newly added component has?
If global (centralized) library is used directly it will be the latest version. If the project has its own library to which the footprints where copied in the beginning from the global library, it will be the same old version than the other 0402 R. Which option is better? I don’t know. It has to be taken into consideration when the strategy is thought out. If all footprints must be up to date with the latest version, it has to be arranged manually anyways, either by updating the board from the global library after every library change or by first copying from the global library to the project library and then updating the board.
In any case the question
may be different than and separate from the question about using git with a project.
EDIT: I’m probably talking about the alternatives which @pedro has faced in their company.
I am a freelancer. I work for different companies with different tastes. The question was how do we manage with gitlab (I think the same solutions can be shared with github) working in a team on the same projects at the same time.
In the example I explained above, all the projects share many components that are kept in the general libraries.
And the components not shared by other projects are kept in specific libraries.
We could naturally have global libraries for the company. One of the reasons for not doing so is that every project is delivered as open-hardware and we do not want to publish more stuff than the needed for each project.
With other people I manage libraries in different ways. Other solution is having a person in charge of the library who is the only one allowed to add symbols, footprints or 3d models.
But this is not the subject of this thread anyway.
Thanks to all for this very interesting and constructive thread! I guess there are indeed two things I wanted to discuss, project handling and library handling. That is probably to far fetched a topic. In our case we are two companies with several developers each, so @pedro it would indeed interesting how you handle that with yours.
As for the actual collaborate work on a project:
The layout itself is usually a relatively small part of the work and in the past it was done mostly by one person. Most of the work went into creating the schematic, symbol/module creation and BOM (oh, how I love that part).
As I see it, the schematic can be broken down into parts that can then be done in subsheets by different persons. In this case, there is the question of whether to use hierachical pins or global labels. In the latter case there would have to be some coordination on the existing/needed labels.
A useful side effect of this tactic is that the subsheets could be hosted as standard circuits in other repos and recycled in future projects.
A central library would have to be used and updated regularly. @Rene_Poschl has a good point including the lib_table in the library repo.
At some point it will come down to only one person being able to work on the project at a time. The most important question to me seems: How can be established, who is currently working on the files? (I think @Rene_Poschl adressed this with the lock files)
I am really looking forward to watching those videos @ChrisGammell mentioned!
-Our project specific library with 3 subfolders too:
Symbols -> project.lib project.dcm
Modules3D
project.pretty
What do these libraries contain as opposed to the general libs? Libs that were created in the course of the project? Are they later integrated into the general library?
We have started this year with this lib configuration.
Before we had specific project libraries for each project. For example, the 0402 resistor was included in all libraries. Now the 0402 resistor is in the common library.
The work of copying all the needed components every new project is tedious.
As I mentioned above we deliver each project with its own specific libraries.
Basically the project libs contains the new components created for that project.
What will happen when one of the new components will be used in a different project in the future? This case has not happened yet, but it will and we’ll need to decide how to manage that situation.
@Rene_Poschl has mentioned something basic: iron discipline. A coordinator or project organiser must distribute the jobs.
I prefer hierarchical pins to global labels. But in a project we had with about 30 sheets and subsheets we had to use global labels. Here the work of the coordiantor is very important.
The schematic with subsheets, footprint library and 3d models can be done in parallel as far as only one person is working on a single sheet, footprint or 3d model. Since they are independent files, merging is not too risky.
But the symbol lib is more critical since it has only one file. OK, two files but the .dcm is not edited by the user.
I believe he meant singular “our general library” where they put their “normal/common” stuff (standardized footprint 0603, 0805, etc for passives, they may always use the same polarity protection circuit, so the same MOSFET, Zenner, etc. and so on, same uController perhaps).
But I’m just speculating from his answer, I really do not know.
This is the way we worked before. I have some collections of symbols and footprints, the most important one the official lib.
When I need a new footprint I search the official lib and then I modify the footprint for my needs. For example, I move the silkscreen to Eco1 for smd components and Eco2 for through-hole components, arrange the text size, etc.
If the footprint doesn’t exist in the official lib, I make it from scratch.
I have managed collaboration on a few designs, one project had multiple boards. How I organized them was to have a folder with all the shared common components in them, like was recommended above, and to point the project files to those folders. It is a little tricky to keep people from putting in components from the general library, that could be potentially managed by a script on check-in.
I also used folders for assets like custom silk graphics in each folder for each board.
For the collaboration part, the main hard part is keeping someone from editing the same file as another user at the same time. It was never a problem to have the files open, but had to be sure to not save them. At least with versioning, we could roll something back. If you had a way to lock the files during use as someone suggested above, it would solve that issue. In general, we had pretty clear jobs to do, and those were deliberately breaking up the work to ensure that people usually didn’t both edit the same file at the same time.
Or a round of everyones favourite beverage at the next organization wide evening activity. Coffee or pizza could also do the trick.)
Or do you want to go through everyones local files just to find that out?)
