I’m using KiCad with git, and I’m sure that many others use some kind of version control too. Now KiCad’s text-based file format looks very promising to be able to not just store the various versions, i.e. like one would do with binary blobs, but to actually merge branches and so on.
I suppose one can’t really expect to be quite as flexible as with source code, for example, since there will be more consistency constraints, i.e. reference designators are (usually) assigned automatically and so on, but I would love to be able to, for example:
Have a project with a master branch, call it “master”.
Say i want to add some features for prototyping. I branch the project in git, call that branch “Proto”. I add, for example, an extra debug serial port connector to that project and commit that to “Proto”.
Then I realize there was a mistake in the main part of the system, so i check out “master”, and correct it. So I might’ve just changed a value, or I might’ve removed or added a component or rerouted one. I commit the changes to “master”.
I check out “Proto” and merge the new changes from master with git. Since the changes only affect the part that I haven’t edited in this branch, the merge should succeed without conflicts, and most importantly produce a valid schematic, maybe even merging the PCB layout, as long as the changes are geometrically separated.
As it stands, at least one thing complicating this is that the power symbol names change in all sheets (when using hierarchical sheets) when doing certain changes, and I’m not sure if it’s consistent to merge schematics with power symbol names assigned at different times. Overall, I guess the underlying requirement would be that the schematic file is generated deterministically from the on-screen representation, including things like internal net names, the order in which various parts are written to the file, and so on. Without a thorough investigation, just casually looking at diffs when committing, I get the impression that it’s not currently far from that?
So, my question is:
Can I do this to some extent as it is? What merges would be “safe”, i.e. that the resulting schematic file is a valid KiCad file without any surprise defects lurking under the surface? How about the PCB?
I have used github for a while to share KiCad projects. Unfortunately parallel changes do not really work. Even simple changes lead to a cascade of changes in the file, which git can’t merge sensibly. Sometimes the merge will be ok, but the only way to be sure is to look at the diffs with a text compare.
Compilers are usually very tolerant of bad input, so if there is gibberish in the input it will fail gracefully. Programs like KiCad are generally reading data they produced, so I would not rely on tolerance to bad data.
The only safe way I have found is to do “exclusive” changes, i.e. if you branch then “freeze” the master until the branch is merged. Git by its nature does not do exclusive, so you have to enforce such rules yourself.
A future feature of KiCad, I guess you mean? Not sure that would fly, basically means creating a whole version control system.
There was some discussion recently of a visual diff for Kicad files, this would help identify changes at the top level. That’s probably a lot of work, unless we can reuse existing GUI code (either from Kicad, or external). Reading the files is relatively easy, writing the rest of the app is the time consuming part.
Yes, I mean a future feature of KiCad, but not by creating a whole version control system. Rather, exactly as you say, making KiCad more tolerant of the kind of output produced by git merges, and on the other hand, producing the kind of data that can be merged (deterministic output to avoid spurious changes, each line is one logical unit, and so on). So leave the version control to whichever system is whoever’s favorite, just make the file formats compatible.
I think that could be quite a unique selling point for KiCad, since as far as I know, no EDA software does that?
The visual diff would of course improve this yet much more, but I do think that this feature could be useful even without it.
I think it is an interesting idea, I’m not familiar with other EDA s/w so don’t how they handle it. I know the guys at work are often cursing our system…
A fundamental issue is the way data are stored. In programming languages, the data is a semantic representation, which the compiler converts to a sequence of instructions. Kicad stores things quite differently. If you look at the .sch files, the data in there is simply a sequence of graphic elements - it’s like a list of instructions, without semantic information. When the net list is generated Kicad works out what is connected to what simply by looking at the x,y coordinates of the graphic elements.
So I think it would require a complete rethink and rewrite of how data is stored. I don’t imagine the Kicad devs will be keen on that. If other EDA software doesn’t do it, then maybe it is either too difficult or simply not needed? I think you would have to demonstrate the need for a collaborative workflow first.
In my experience of open source hardware, both electronic and mechanical, the design process really doesn’t work well with multiple designers for a single item - a PCB or mechanical assembly. If there is a well-defined interface, then you can work in parallel but isolated, and all the projects I have worked on have managed it like that.
I’d personally at least use this for more than collaboration. Look at my example above, that’s valid without any collaborative element. I would imagine I’m not the only one who might find use for having an “instrumented” prototype/testing version of a device, and then a leaner production version, and then it would be useful to be able to import changes from one version to another, without having to redo them separately in both.
Based on how few changes there tend to be in a commit, just from eyeballing it, I think it might be possible to hack this into the current data model. The developers would just have to make sure that the graphic lines are always stored the same way, for example sorted by y-coordinate of the initial point, and such that the line-by-line logic of git doesn’t break what semantics there is (i.e. there’s always a guaranteed line break between the description of each graphic element, to prevent the merge from conflating to separate lines).
I’m not a professional EE, so this is just a hunch, but I get the feeling that EDA software is a very conservative thing? Maybe this, combined with the fact that usable, popular, open source DVCS’s are really quite a recent thing, could be the reason that it hasn’t been done before?
I think it is more to do with hardware design being fundamentally different to software. Hardware design does tend to be conservative, and that is partly there is a well-established way that works, and there is a big cost of screwups. Software design is still trying to figure out how to design things, the latest flavor of the month is “agile methods”. There is a low cost and potentially large benefit of trying new things in software.
In all the companies I have ever worked for, there is only one schematic designer per PCB, although they might take advice and do peer review with other engineers, they is one person responsible for changes. OTOH, I’ve worked on software projects where there is a single binary but hundreds of developers.
I think both methods have evolved for good reasons, it’s not due to the availability of tools. I would say that the development of tools is driven by the methodology. Git for example, was specifically developed to help with the development of Linux. Before that, tar was used - which was designed for streaming files to magnetic tape.
I have worked in a few places where there is more than one active contributor to a schematic or a PCB layout. There are a few circumstances where it is incredibly useful:
Using Mentor Expedition / DXDesigner, the simplest way of schematic split is that an engineer signs a schematic page out. That page is then read only until checked in. The PCB package has a number of ways of splitting work: Assign regions of the PCB, Assign layers, or free-for all.
The circustances:
(i) Separate teams for power, baseband & RF on a cellphone.
(ii) International round-Robin, where a designer in the UK can hand over to someone in the USA, who then hands over to a colleague in Hong-kong, before hand back to the UK in the morning. At each hand-over, the work done gets reviewed before more is done.
(iii) Incredibly complex PCB for a digital video system, where the DSP for audio & the FPGA for video sections were able to be routed simultaneously.
@bobc: so you agree that the lack of version controllability as a feature might be partially due to (well founded) conservatism, it’s just not in the EDA software side only?
As you say, a big company is unlikely to take the risk of trying collaborative PCB design or versioning a single PCB with branch merges, since mistakes can cost a lot. In this sense, I think KiCad could be perfect platform to try it out and see how useful it would be in the real life, as it’s anyway used largely by hobbyists and other people who can afford a bit more risk (remember that even CERN still officially supports only Altium). Also, this is not a feature that should cause anyone to shy away from it, since you’re not forced to use version control, or if you do, you’re not forced to merge changes between branches if you’re afraid of the risk.
@Martin_Layley: so you think that the collaboration side of this could actually be useful in large commercial projects?
Like any other project, you still need a design lead who mantains overall responsibility. The ability to assign sections of work to other people can speed up certain design/layout tasks.
The alternative is for the team to hand draw on paper and then pass over to the single CAD engineer. Worked like that at Samsung 12 years ago.
I think the idea has merit, and there is certainly some chicken and egg involved. I can’t disagree with people who say it would be useful, and I wouldn’t object if Kicad supported it. Of course, it is not me who needs to be persuaded, it’s the dev team
I understand eschema is being overhauled as we speak, so now would be a good time to get ideas considered.
The theoretical aspects pique my interest. The diff/merge done by tools like git is primitive, it just compares text line by line, it has no knowledge of the syntax or even what type of data it is. Yet, that method still works quite effectively for software. So the questions are why is that, and could a similar thing be done for hardware designs?
I have some ideas. I will have to think on them while I watch the football.
Collaboration on Eschema should not be too difficult to arrange on a per sheet basis, as Eschema provides quite a nice set of hierarchical tools, and the sheets are in separate files.
I’d be very interested in this as well. I’d think one would want a graphical diff, though, which would really be a separate tool. Something to compare changes between schematic files in a graphical manner.
However, I believe that git allows the user to supply a “merge driver”, which replaces the built-in line-by-line merge. So, in theory, it would be possible to write a program to merge KiCad schematics (or another one to merge KiCad PCBs) which could be used as a “merge driver”. This could be a separate program and wouldn’t have to be part of KiCad.
As for the schematics, it would be great if someone would build binaries from plotkicadsch. Then the rest of the work flow would be practically same as above. Another option would be if developers implement command line interface for eeschema, which has been proposed before https://lists.launchpad.net/kicad-developers/msg06052.html
This coveres only diff functionality, which is based upon svg files. Merging is completely different beast as @bobc already pointed out so I am not even thinking about it.
As for development, concurrent development is doable in eeschema if you limit yourself to separate hierarchical sheets. I don’t know how the pcbnew workflow looks like, but I’d like to hear some experience.
I have done a ‘Visual Diff’ for PCBs. (https://github.com/Gasman2014/KiCad-Diff) It takes commit references and performs a visual diff between them (or between current version and commit reference). The output is presented as a webpage, split by layer with the differences highlighted in red (removed) and green (new). Happy for anyone to come up with a less garish .css!. To make a meaningful text diff to back annotate which actual component has changed would require a lot more work. I think you could do it by parsing changes in the s-expression back up to the source component in the tree.
If there was a python interface to eeschema, it would be easy to do the same for schematics. I have tried to do this using janvila plotkicadsch but with limited success (plotkicadsch is still in development). I have only got neo900 to run on Linux.
I was about to suggest something similar with KiCad. It should be possible without big issue to work in separate schematic pages. I believe only that schematic page file will change.
