A brief(ish) history of inconsistencies

Software
1

A question has been repeatedly asked in various guises: “Why do schematics and PCBs work differently, especially for graphics?”. Asked genuinely, this is an interesting question about KiCad.

For those who actually do find interest in technical and historical reasons for this, and I hope there are some around, I’ll try to outline an incomplete explanation. This is all according to my own understanding, which is not complete, but I myself am interested in the “why” of these things only inasmuch as it improves work on the code today and I find this understanding sufficient for now. However, I have worked in the area recently, so I have a few details in mind. This is mostly “recent” history from about the last 10 years. There are 22 more years before that, which would be an interesting journey for someone else to document. You could, quite literally, write a book about this stuff. If I get things wrong, hopefully I’ll be corrected.

Most of the things here also apply to other areas than drawing tool inconsistencies, but they happen to be particularly good examples.

  • Once upon a time, there was something now called the “legacy canvas” (at the time it wasn’t called that, obviously) This was the core of the drawing system in what is now KiCad, and dated right back to the inception of the project in 1992. By the early 2010s was suffering from scalability issues and it was hard to add features without breaking other things. It was also slow (without graphics acceleration) and didn’t handle large boards well. Remember, 1992 was only 7 years after the first release of C++, the year before Visual C++, 13 years before Git (and 8 years before SVN, but 2 years after CVS) and the very same year as wxWidgets and a single year after Linux. Ask Jeeves would be 3 more years. ADSL was 6 years off and 2G (digital!) was brand new. Things in software and electronics design and manufacture (6 years to RS-274-X Gerbers) were really rather different then.
    • In particular, enormous classes called FRAMEs (which were only renamed in 2011 from things like WinEDA_BasePcbFrame, which betrays an earlier project name) contained nearly all the state of the program, including active tool state (e.g. when drawing a line, what the fixed end is, what the free end is and if theres’a 45-degree constraint, etc. etc. etc.). Those who have worked on large programs will know that state management is THE key thing that permits scalability and maintainability. Sufficient in the 90s and 2000s, the system was straining under the functional additions added rapidly in the early 2010s.
    • FRAMEs still exist and are still very large, complex, important objects, but contain far less complexity than they used to have.
  • Starting around 2013, CERN engineers added what we call the TOOL framework (TOOL being a class name in the C++ code), which is a way to encapsulate the state of a tool in a way that is more modular and less likely to break things. This is why you rarely see one tool badly break another these days, but it was super common a few years ago.
    • This framework is based on the concept of event-driven coroutines, inspired by game engines in particular, though they have many other uses from academic interest right through to industrial robotics.
    • This framework was complemented by a complete rework of the drawing canvas to use what we call the GAL (Graphics Abstraction Layer). This is a way to draw things that is more efficient, scalable, maintainable and flexible than the old way.
  • At the same time, the “modern” C++11 standard was adopted in KiCad after much consideration in 2016, which enabled enormous amounts of modernization in this process (relevant commits: 2d4845ddae, 27a10e8597 and many more). This process continues, and there’s still a lot of things to improve, for example, safer memory management. But the process is a bit like replacing all the bones in a person, but in such a way that they can still do a daily 5km run.
  • The legacy canvas co-existed with the GAL for quite some time and features were gradually ported over one at a time over several years. This was often immensely frustrating, as it only added to the complexity of the codebase in the short term. Many times a tool was removed from the legacy framework to sidestep that technical debt, or make improvements only possible in the TOOL framework, and there were complaints that it was “broken” because the GAL didn’t have some other tool and switching was annoying. Chicken and egg.
    • A huge amount of clean up was done in 2019 to remove scads of legacy canvas code and unblock more GAL-enabled features. Look for 2019 commits with “legacy” in the commit message. This process continues to this day.
    • Example: the array tool, added in 2016, used to use a quite complicated system called ARRAY_CREATOR, as it had a legacy and GAL version with a polymorphic interface to support both in a unified way. The legacy version was able to be removed in 2019. However, it remained an oddball system until just a few weeks ago when it was finally moved into a dedicated TOOL framework tool.
  • Pcbnew was ported to the GAL first, partly because the PCB editor was most hamstrung by the legacy canvas, partly because the schematic editor was “good enough” at the time, partly because the new router required the TOOL framework and partly because people simply preferred to work on pcbnew, which is self reinforcing when eeschema was much more frustrating to develop for.
  • The schematic editor was ported to the GAL later. So this means that new tools and features were added to pcbnew, with its new framework and drawing system, and not in eeschema, which was if not in limbo, at least a bit less loved until the open-heart surgery could be scheduled. Thus, there was never a time where the two programs were in perfect parity. Pcbnew TOOL framework work started around 2014, but eeschema took until 2019 (relevant commit hash 5f303f7b).
    • I can tell you with perfect certainty that the development team was absolutely desperate to get eeschema ported to the GAL reap the benefits. It simply just did take that long to do it right while also keeping KiCad working and shipping. Also during this period was the move to s-expression formats, which was a long painful and extremely necessary process too.
  • On top of this, the two programs had always had two distinct categories of objects that can be placed on the canvas: in schematics, they are SCH_ITEMs and in PCBs they are BOARD_ITEMs. Both inherit from EDA_ITEM, but this means that a class that expects a BOARD_ITEM can’t be used with a SCH_ITEM and vice versa. “Hoisting” the relevant polymorphisms up to EDA_ITEM can be tricky and takes time to carefully consider and then work that into all the places that need to be updated throughout all of KiCad and check that it doesn’t break anything.
    • In particular, only quite recently (late 2021) did a common shape class EDA_SHAPE become possible, which is a core prerequisite for having a common drawing system for both schematic and PCB. Relevant commit hash: a41944020.
  • So this is why eeschema didn’t get the same “live” drawing tools as pcbnew. Example: when the ARC_GEOM_MANAGER and ARC_ASSISTANT geometry tool code was added in 2017, there was simply no framework for them to go into in eeschema, even if they could interact with the eeschema objects, which they couldn’t.
    • Because there is no equivalent mini-framework for the interactive drawing tools, it was also not possible to implement the same bezier curve routines in eeschema as in pcbnew. Rather than simply leave beziers out in eeschema entirely, a simple version was added as a stop-gap until the entire interactive drawing system can be unified (likely in v10)
  • Another very complex area of code is the point editors. First added in 2014 as part of the TOOL framework initial concept, only pcbnew had one as eeschema wasn’t ready for it until 2019. So both because they are very different ages, but also because they deal in completely different types of objects (e.g. EDA_SHAPE unification not until 2021), they historically haven’t shared a lot of code but instead been parallel systems until literally about 2 weeks ago (relevant commit hash: 87cd0a74f2 and the 10-15ish prior to that). This has already brought unification of, for example, the reference image editing mechanisms.

Those are some of the technical reasons for the way things are. Or excuses, depending on your perspective.

Something else which is sometimes mentioned when this question comes up is that pcbnew and eeschema may now be on the same technical basis (TOOL and the GAL), but they are still two different programs with different user interfaces and user expectations. This is a much more difficult problem to solve, as it involves a lot of human factors and design decisions. It’s not just a matter of copying and pasting code from one to the other, interfaces have to be carefully considered so that code can be shared between the two programs without breaking the user experience of either or introducing unacceptable complexity or limitations.

For example, drawing a wire in eeschema is a very different operation than drawing a line in pcbnew, though both are “just lines”. You can look at the classes EE_GRID_HELPER and PCB_GRID_HELPER - these are very different not only due to historical and technical reasons but also for functional reasons. It’s hard to find a common ground between the two that enables, rather than hinders, the user and developer experience and doesn’t break things. But we’re getting there. Sometimes it’s simply better to have two parallel systems, and at that point it requires active effort to keep things in sync to the perfect degree. If we slip, or one side gets an upgrade, but it’s not clear how the same upgrade applies on the other side (e.g. a PCB line drawing improvement may not conceptually map well to wire drawing), the two programs can drift apart.

Notably, what is missing from the above is a big-bang re-architecture leading to periods of complete unusability, invasive shiny new hotnesses that are added in a hyperactive flurry and then abandoned when the champions burn out, breaking changes and project stagnation. Everything is incremental, and the program is always supposed to be working. Sometimes that can feel slow from the outside looking in, but in the longer term it’s still faster than the boom-rewrite-burnout-stagnation-revival cycles that many other projects go through. And at something like +200k/-85k lines of C+±only, non-third-party diffs from 8.0 to now, there’s actually a lot of velocity in KiCad. I have been in projects where there’s code churn for no gain, and KiCad is not that. Those are 200k lines of useful, directed work. And we’ll do it again year after year.

Sometimes, an effort is made to fix something trivial and that uncovers a whole mess of other changes that are needed. A good example is adding the ruler tool to the footprint picker tools. This sounds simple, but it actually requires adding an entire TOOL framework to the relevant UI elements, even though they are not FRAMEs, and their parent FRAME already has one, which requires some re-architecture.

Sometimes, it’s not a technical issue, but still thorny: we know hotkeys aren’t very consistent, and changing a single one is trivial, but coming up with a new hotkey set is both very tricky and guaranteed to spawn a lot of complaints whatever happens. Sometimes thing are just left as they are because there’s the cost-benefit doesn’t work out for anyone, team member or not, to undertake it. Sometimes, it’s just not worth polishing things to a mirror shine when you know there’s a lot of work on the horizon that will change it all anyway. Sometimes that change doesn’t come and a bit more polish may have been ideal, in full-HD 10-bit retrospect.

Hopefully that helps explain some of the technical reasons why things are the way they are. If you found that interesting, there are a handful more resources below, and perhaps they’ll spark interest in getting involved in the development process. And if not, you can still see that there are at least reasons for things.

TL;DR: the code is very complex and it’s not been possible to produce consistent tools up to this point due simply to the scale of the task. But it’s been a goal for ages, lots of work has been done and now it’s not so far off. It’ll never be 100%, as PCBs and schematics are fundamentally not the same thing.

43 Likes
2

Excellent writeup, John!

Thanks for doing that.

3

Currently, there are 81 issues marked as Technical-Debt, and 41 of them are open.

4

Great description, very understandable presentation,
Thank you

5

@johnbeard -
Thanks for taking the time to share this.

I’m grateful for the time and effort that the development team has invested, and I am particularly thankful that Kicad is cross-platform. I’m sure the whole system would be dramatically simpler if it didn’t have to support 3 OS’s (and all their variants).

6

I’m actively using KiCad since something that could be considered KiCad 3 in today’s terms (naming scheme was different back then) and I can clearly see the great progress done. It’s a great deal of work, to successively improve such an extensive project, and still keep it usable during these changes. And this organic growth will surely bring many legacy issues and inconsistencies, unlike the system built from scratch (but maintaining all the experience gained during the process). KiCad dev team is doing great, much better IMO relative to such high-profile (and definitely massively greater dev-base project) like LibreOffice for example. Thank you for your great work!

2 Likes
7

Wow, thanks for that very detailed writeup. I think progress has been nothing short of amazing.

While cross-platform support has costs, I think the benefits are very worthwhile. KiCad would not have as much of a user base if it were only confined to one platform.

1 Like
8

Thanks for the background, but ouch my head hurts. Reminded me why I didn’t become a programmer. Thank you for all the hard work you all have put into KiCAD. It’s been wonderful to see how far everything has come in the past few years and glad it’s improved to the point where a hack like me is now able to use it. :rofl:

9

Thanks much for this! Bravo to the team who have worked on kicad through the years. There was a time when I tried to sort through the code base but quickly became overwhelmed so I can appreciate the efforts here.

I will say though that one of the strengths of kicad is the use of text based files, especially the schematic files. This is a real enabler for the casual programmer or someone who wishes to implement some very specific functionality without tampering with the kicad code itself.

Thanks again!! Kevin in Maryland

10

Thank you to ALL the developers both past and present who have contributed to this project. I only wish I could contribute to the code. Maybe one day.

11

You see this exact thing in every aspect of human society when you look.

Someone makes a decision about something that makes perfect sense at the time.

Later, someone makes a decision about something new and is constrained by the decisions of history.

Loosely speaking here, and I know it’s not 100% accurate, but…

Train lines are a certain width pretty much globally because Roman bridges were a certain size, and greater Europe saw the opportunity to use existing infrastructure for new technology. They made the first trains, others bought.

Roman bridges were a bit more than the width of 2 Roman carts so they could pass.
Roman carts (and most likely everyone’s) were basically the width of 2 horses/oxen walking comfortably side by side
I seem to recall some constraint around axle length back then too possibly because the differential or in-wheel bearings weren’t invented yet meaning the wheels were fixed to each other and would have to slip in turns.

So the end result is that if horses were twice the width, or half, the latest bullet trains likely would be too.

Now look down at the keyboard you’re typing on.
Qwerty was a decision made to fix problems with typewriters jamming by rearranging the letters so that the very real levers statistically would be less likely to hit each other as fast typists typed. I think there was also some consideration given to relative finger strength.

It’s a completely irrelevant constraint now, but because of that decision, and the billions who only know qwerty it’s practically impossible to switch back.

Even our letters were shaped by multiple cultures and technological constraints. The letter A has been traced directly back to a hieroglyph of a bull with horns. It’s upside-down now, and at some point was shaped by the chisels used to carve it into stone, but it it the same logical representation. M is from a glyph/rune representing water. It is almost unchanged, but is definitely sharper due to the chisels.

Our cities were shaped by their times too. Goat tracks became roads, shanty towns became historic buildings no one can demolish for any reason. Wars required walls, moats. Istanbul and many other walls are unnecessary now, but still there.

If you look at the evolution of the x86, I see it as fairly similar to the human brain. It started out as 8bit. Basically a lizard brain. Eat, fight, fornicate, sleep.
Over the years it’s added layers, and is the modern 64bit human brain, but if it’s stressed in the right way, the upper level code crashes and it reboots back into the legacy lizard code. I’ve read that hiccups are evolutionary leftovers from fish. It’s how they pump water through their gills. All mammals have their vocal nerve take a long route under their aorta and back to their throat - which is hilarious in giraffes. Lag!

Untrained people have only 3 reactions to mortal panic. Fight, flight, freeze. Default programming.
Side note… It’s amazing how every pilot who has a survivable crash landing is usually asked by an interviewer if they were scared and they pretty much always blink a few times as though it’s something they never considered, and then state more or less “the training kicked in and I did what I had to do”. It happens to many other people I’m sure, but they’re not really interviewed as much.

Anyway, don’t feel bad.

Legacy decisions made under constraints which no longer exist are literally everywhere.

It’s evolution and you’re part of it!

5 Likes
12

What a great write-up.

I’m very grateful to all the KiCAD contributors.

I’m not a professional pcb design engineer, but from time to time I have a need for and easy to use robust design tool for our simple projects. KiCAD has never let me down.

Sure, there are some idiosyncrasies, but once KiCAD has ‘trained you up’ there are no issues. I can’t sympathise with new users who complain that it doesn’t work the same way as the proprietary software they are used to! If it doesn’t suite you, don’t use it.

I echo @djsbriscoe , I wish I could code and contribute.

I’ve also learned so much from this forum. Such a friendly bunch!

Thanks again

Mike

1 Like
13

As someone who has complained about the KiCAD UI, I want to thank @johnbeard for that excellent account.

Like most of us, I’ve been building, programming and using computers (PCs, mostly, in my case) for decades, and I’ve come to the view that UI design is extremely difficult to do well, and even more extremely difficult if you’re a software engineer. :smile:

I think this is true for at least two reasons. Firstly, if you’re the person who wrote the program, you have an intimate knowledge of every part of it: how it works, why you made the various design decisions, and so on, and that level of understanding makes it almost impossible for you to see the program through the eyes of a “mere” user. You can try, but it’s almost impossible, I would suggest.

Secondly, pretty well all software engineers are, by nature, “geeks”, and geeks just see the world differently from “normal” users. Something that is obvious to a geek might be completely opaque to an average user.

I think the fact that Microsoft and Apple both invest big money into their usability labs shows how non-trivial this whole area is. Obviously, most open source software just doesn’t get to benefit from the level of UI expertise that MS and Apple have spent some serious dollar on.

Thus I recognise that it’s probably unfair to complain too much about the UI when using open source programs. It’s probably fair to say that the open source model has benefits and disbenefits, and UI polish is one of the areas where open source is at a disadvantage.

I’m writing all this because I feel a bit bad for complaining about it. :smile:

3 Likes
14

I appreciate the effort you’ve put into explaining the effort put into building and maintaining KiCAD. Thank you!

1 Like