Since kicad files are just text, it occurred to me that you should be able to do a merge just the same way as you do it for code. But it turns out that the code is structured really non-optimally for such a change to work out easily. Well i suppose that is to be expected since its machine generated. Even so the code could be slightly differently indented for a easier time recognizing the right things.
Though it should probably be easier to do this with a dedicated merge tool. Does anybody know if someone is working or has worked on such a tool?
It would be great if 2 people could somehow be able to work at the same schematics semi independent of each other and be able to merge it just like programmers do in version control systems.
@joojala I am having the same problem.
I do use git to manage my projects. But as you said, when a collaborator push his change, there are tons of unnecessary change in the file, makes it doesn’t make sense to view the git diff.
Lots of them are re-annotation the parts reference, which still somewhat acceptable.
An totally not undertandable is something like this, the path is totally hash number, so not suppose to be human readable.
For me, that looks like a real path to a folder, but I do not know where that is.
I know debian people have spent lots of effort to make it binary reproducable ReproducibleBuilds - Debian Wiki .
I hope one day, the kicad file do not have to use this kind of hash.
I know that there are tools for this in existence. I haven’t ever collaborated over a PCB design myself, but the below links seems fruitful.
Things you can do concerning schematics: work in your own schematic sheets and may it be obvious to use verbal communication to prevent working on the same items. It will propably not prevent all merge conflicts.
Concerning board design, while typing this answser the following came to mind:
Let coworkers create their own board designs and let one guy merge his work in the main project.
For example:
Your ‘thing’ needs a buckconverter. You allocate this task to your coworker. He makes a schematic sheet either with global or hierachial labels. And he also annonates it.
Than he copies that schematic (with annonation) to a separate project and he places all components, routes the board, add via’s etc. (I would only let ground plane out, but small details…)
Than you can manually group and copy his work and paste it in the main board file. If you do it right there won’t be a mismatch in annonation. (By grouping it, you cannot accidently forget to drag a trace or via. Small detail…)
That is indeed a bit FUBAR. Reminds of that Beckhoff twincat environment. Opening and saving a project alone without doing anything can already result in a change in the main file.
It is fundamentally not trivial to do this, because a text representation of a schematic or board is very different from a text file containing source code.
The source code file is basically “one-dimensional” – that dimension is position in the file. For example, if you identify a chunk of text containing a code function, that function’s position in the file can be described along only one dimension (number of characters from the start, for example). When dealing with only one dimension, it is rather simple to consider the problems of merging together multiple changes to the file from different people. This is a bit of a simplification, but hopefully my point is clear.
If you are looking at a file that describes something like a schematic or board, this is a higher-dimensional data structure. The objects represented in a file have at least two spatial dimensions, and maybe other dimensions representing other structure. Just because the file is readable to you as a human and appears as text does not mean it represents something that makes sense to merge together the way that one-dimensional text files can be merged together.
Because of this, it does not make sense and won’t ever make sense to use text-based merge tools to merge design files in the general case. There are specific cases where it is possible: that is, cases where the differences to be merged are isolated to a single dimension, or do not require structural changes to the file. But a merge tool that can only handle certain scenarios is not very useful.
A general-case merge tool for this kind of higher-dimensional data needs to be purpose-built. There are various approaches, none of them trivial.
Having said that: Merging and version control are often conflated but they are different concepts. It is quite possible, and in fact very common, to use version control systems without any merges at all. In my experience, this is the way version control is used with schematic/board files from KiCad and other tools.
Well, yes its harder. but it does not have to be this hard. A dedicated merge tool would be nice. I mean even my 3D CAD is capable of doing a 2-3 way merge although its not really meant for concurrent working but rather model ideating, but still.
Yeah i am aware that locking versioning system are a thing, after all i have used SVN and perforce for years. Also most CAD PDM systems are locking as well (PDM is mostly also sort of a versioning system).
Im ok with restricting things to sub sheets, just as long as the main sheet could be updated by merging.
It doesn’t seem like you understood my first post. No, changing the indentation would not help. Yes, a special tool is necessary. Just because KiCad files are human-readable does not mean they can be merged like code files using a 1D-text merge tool.
Yeah i understood what you said. and im still of the opinion that your not entirely right either. I mean i understand need for a guid for a item, thats fine. But changing indentation a little bit would make the simpler cases simpler.
But reading this post i gather following. A commercial merge tool exists, yay. Nobody is particularly interested in such a tool. So i wont bother with even starting making a dedicated tool. Because in order for me to make such a tool i would need to be able to do it manually as a first step
well, one of the things that makes merging harder than necessary is how the parenthesis are laid out. So it uses a methodology where if one list fits one line then the parentheses are all on that line. But if subsequently you change things and it does not fit one line the closing parenthesis jumps out of that line.
Also there could be a bit more strategic line spacing especially in second level symbol definitions. so you can easier distinguish them from the 3rd level symbols. But then i did also develop in lisp and i remember the confusion of s-expressions.
But to be totally honest, all i really would need is ability to load 2 files into one schematic and have it in a separate group, layer partition so that i can color them differently and decide what to delete and what to keep, while keeping guids and so intact. Bonus if once duplicate is gone they would turn back to normal.
Visually comparing schematics and layouts is a significantly easier task than determining semantic differences between them. Take a look at these projects.
Yes, i am able to overlay PDF’s, or SVG’s too, no problem. But useful for commit scripts sure.
But optimally in future it would be nice to be able to merge designs if not automatically then it would be better to have the overlay inside kicad so that i can edit as i go. A bit like kicad-Diff but inside the software itself, would make manual merging easier.
If you insist on a tool in KiCAD as such, then here is what you could do:
Use gitpython to navigate to a particular commit that you want to compare to some other commit.
Get the contents of the layout from both commits.
Plot everything from the reference commit to the usual layers.
Plot everything from the other commit to one of the 10 user layers.
Having everything in the same layout, but in different layers should make it possible to compare changes.
Now, this will work for the layout, but not for the schematics. As far as I see, there is no python support for the schematic editor at the moment.
So the central problem seems to be that kicad stores the components in a range tree. This means that when saved stuff may jump around in the saved file. Some applications that are functionally similar seem to store the data in both a spatial tree and in a hierarchy tree. Since they are in hierarchy tree when saved they respond to merging much better since they in fact work more like 2D like in code rather than 3D file. As a result they get other features for free at the same time.
This wouldn’t actually be a huge change nor would int really consume much more memory either.
I did a little test that if I sort all the layout items in the text file, save it and open it again, that does not seems change the pcb or the schematic. It seems the order the components saved does not really matter?
Is that possible to just re sort the text based file before the git commit so it is comparable in the git?
Is that possible to just re sort the text based file before the git commit so it is comparable in the git?
Yes. Even better, like I said, they would just change the data structure so that the data is in both in range tree and a list. This is very easy to implement.
Nearly all other 2d and 3d applications do this double list to some extent or another ( software include maya, 3dsmax, solidworks, creo, illustrator, inkscape, corel etc…). it would make memory consumption a bit higher, but since a schematic has less than a 100 000 components realistically it would use less than a half megabyte of memory.
It seems the order the components saved does not really matter?
no it does not, everything gets loaded in the range tree anyway time to draft this as a wishlist request. Done:
if anybody want to change language of the request, please be do not hesitate to say so.
Ok seems that im not entriely correct the file is sorted. Its just not sorted in any sane way whatsoever. It is sorted by UUID, which while seems like a fine thing for a computational point of view is not really what a user would want to use.
Sorting by name is indeed much much easier for merging. Because UUID might change if you copy stuff then delete it. and rename to match what was there before.
But would be easier if there just was a tree that defines file order then users could themselves order stuff and make sure they don’t edit same thing at same time-
