I’ve been exploring the challenges of scripting in KiCAD and its limitations. To address these, I’ve started experimenting with the concept of remote controlling KiCAD. Currently, I’m developing a prototype that integrates kigadgets (formerly known as kicad-python) with Pyro5. This combination enables accessing the python API through localhost or, if preferred, a local network.
This project is still in its initial stages, but my primary objectives with this scripting method include:
Enabling interaction with and manipulation of KiCAD from separate processes.
Allowing the incorporation of any necessary dependencies into the environment of these processes.
Utilizing advanced development tools like debuggers and Jupyter.
Supporting long-lasting processes that do not disrupt KiCAD, even when they are CPU-intensive.
Creating a clear, typed python interface that aids IDEs in providing accurate hinting and suggestions.
I’m eager to hear your opinions and advice on this matter! Even if you think it’s not the best approach, I’m open to suggestions and learning about better alternatives.
The current SWIG interface will be deprecated and removed in a future version of KiCad (perhaps KiCad 10), so developing a complex wrapper on top of it is probably not a good long-term plan, but if it is useful for you in the near-term, great!
The new API will have a Python interface available and we will be welcoming contributions from the community to extend it once it is made public (current target is for KiCad 9 for initial version). The new API has many of the same goals as your project (allow long-running and separate processes, allow use of normal IDEs/REPLs instead of the limited functionality of the built-in scripting console, etc)
That all sounds awesome - way better than what I’m currently dealing with.
KiCAD 9 feels like a long way away - is it just in a planning stage or is there some alpha I might be able to play with in the meantime? This is something that’s really nagging at my project and it certainly felt like the SWIG API was on its last legs - which is why I started looking for something better.
I have been contemplating that idea for awhile as well. But I hold my hurry to do it since I heard some time ago that the team plan on pulling python out for a more “native” IPC/RPC solution .
I think the value of working on it now, before or in parallel of their work, would mostly reside in exploring what could be done to give a more “pythonic” API to Kicad. And by pythonic I mean something more friendly to work with, that flow easily, and express well what you are doing.
It is actually really hard to craft such nice API!
And in that regard, there are various community efforts that already tried to do that. Kicad-python being one of them.
We don’t necessarily need to wait on the team for that. And it doesn’t have to be all lost work in the end. Firstly, if you consider that the IPC/RPC part is just transitional while waiting for the real stuff, then it’s actually fairly low threshold to make a throw away one that just do the “carry around bunch of data” part, not trying to model, validate, handle edge cases, and all the other hard things.
Secondly, if designed well, then when the real IPC finally comes out it could be just a matter of switching from one internal intermediate interface to another. Especially if that intermediate interface evolve progressively while tracking how the real one develops.
And finally, while this is carried out in parallel of the team work, I guess it could also help them as being sort of an experimental playground, and give them inputs about what the real IPC layer would need for such pythonic interface to work well.
(Additionaly, my thinking on it so far was that this internal intermediate interface could also serve as an adaptater not just to the IPC layer, but also possibly to a pure “offline” (ie. no kicad running/installed) SExpr parsing/exporting (community) lib. You know, just to have more flexibility when it comes to execution modes.)
I think the need for a third-party wrapper is going to increase, not decrease, in the future. Although kicad-python was abandoned by the KiCAD devs, it was the place (third-party to KiCAD) to make version-independent API. KiCAD v6 introduced many changes in SWIG. All my action plugins broke, so I took up the kicad-python project in 2020.
My lab uses kicad-python heavily. It doesn’t need much maintenance, just a few updates when v7 came out. As I’ve been receiving more interest in the project now, I’m finally doing a push to actually package, deploy, test, and document properly. I will make a forum post for feedback on the kigadgets beta in the near future.
Github
Note, the “main” branch is still the old kicad, while branch “4.99” is the repackaged/renamed kigadgets in beta.
I really appreciate this discussion - it helps get an idea of IPC interest. IPC for KiCAD hadn’t occurred to me before @mawildoer suggested it. I can chip in some thoughts about kicad-python status with the topics that arose above.
Bridge to new python
Suppose a new python interface were introduced in v9. I would then interpret “long term” as v8 becoming unsupported, as opposed to v10 being released.
Any scripts/plugins out there would still have to support SWIG until v8 becomes unsupported, perhaps splitting projects into <v9 and >=v9 projects, or they simply won’t migrate to v9. Also, I think it would be reasonable for developers to anticipate unpredictable changes in v10, as the new python strategy moves towards a stable form, and also because of KiCAD’s track record on python support + roadmapping. That is fine though! It’s par for OSS! All it means is that developers need other ways to ensure stability and compatibility, such as version-independent wrappers.
Whatever the transition looks like – pybind, manual API, etc. – kicad-python will wrap it just as it does SWIG. As of now, that is my laboratory’s plan to ride out any future transitions: build everything on a third-party library that gives stable API, never adopt any built-in python API because kigadgets will update to use that API when appropriate.
This is exactly the type of migration that we don’t plan to do.
Using the API
I like the API that has come together. There is an example action plugin for a mousebiter in “examples/action_plugins/mousebite_kigadget”. The mousebiter is 180 lines of code incl. comments. It is pythonic in the sense that an English speaker can kinda get an idea of what is going on at the same time that there are excessively complex list comprehensions. Such as
is_vert = lambda seg: seg.start.x == seg.end.x
edge_dwgs = [dwg for dwg in board.drawings if isinstance(dwg, Segment) and dwg.layer == 'Edge.Cuts' and is_vert(dwg)]
Pythonic.
I also like environment and entry-point handling that lets it have 3 entry points: API/CLI/GUI. The API entry point can be used for continuous behavior testing. The GUI entry point gets a nice button and dialog. All three use the same 180-line backend. These are demonstrated and checked for correct behavior under “action_plugins/tests”. You can run tests with pytest examples/action_plugins/tests.
Using external IDEs/REPLs with the Pcbnew GUI
This is done. See “examples/action_plugins/onepush”. It is a bit hacky compared to IPC but works fine.
IPC
Having a development-cycle-synced package running both within and outside the GUI is the easiest way to do IPC in my opinion. The problem has been solved before, for example, lyipc does the same thing for a different CAD application (KLayout). It is not a particularly difficult addon. lyipc is 600 lines, although most of that heft is for things now provided by Pyro5.
Something that might be tricky for a built-in IPC is to have a KiCAD v6 window on one machine controlling a KiCAD v9 window on another. Not sure why you would want to do that, but it would be trivial for a third-party IPC implementation.
@mawildoer’s PR above I think is pretty far along. If there is broader interest, then we could potentially prioritize and support IPC/RPC for v5-v7.99 in a matter of weeks, hopefully have it battle tested in the wild in some months. Do you think there is broader interest in this? Would people be interested in IPC clients for non-python languages?
Over the years, I’ve seen a number of kicad-python-like repos laying dead on the side of the road. I’m glad kigadgets seems to be active and going forward. The ability to 1) span multiple versions of KiCad without plugin revisions and 2) program in a Pythonic manner would both be big wins.
Heh, I shall defend it by stating it’s a evolution of lessons learnt packaging over the last few years. Windows and macOS are the majority of the kicad user base. At the same time python is a nightmare to package on these platforms with it’s own incomplete designed by committee embedding API abandoned years ago because vast majority of python users don’t use it in an embedded into app sense. There are far better languages such as lua for embedding into apps.
No shade intended by that. v6-8 progress has been amazing to use. Seriously, my group designs everything in KiCad even though the organization gives us Altium licenses.
I did get concerned about KiCAD when v6 SWIG changed without documentation. Hence my interest in making kicad-python version-independent. With the updated wrapper API, we don’t stress much about official python roadmap anymore.
Why python: I’ve found python works well for PCBs and ICs. Many people designing hardware are not software engineers. They don’t want to hear about development cycles or new languages. Most have heard of python and can write python plugins/code for KiCad. These plugins can get quite sophisticated if it is a difficult problem. End of the day, they almost never choose to publish and maintain code because there is more hardware to do. We can’t all be MitjaN heroically maintaining 3 parallel releases of plugins.
I can empathize about embedding python. There was a big roadmapping juncture a few years ago with KLayout (integrated circuit CAD) having to do with embedding python. The macos linker was a special headache for them. Ultimately, the devs decided to package python and Qt in end-user binaries but make both of them configurable in source builds. They then deploy Qt-less builds to PyPI - we use those very, very often for CI and other cases where the full application is not necessary. They could release python-less binaries but made a judgement call not to.
I use mostly macos for PCBs; some group members use Windows. kicad-python works on all of them inside GUI. Outside GUI, Mac/v6 is good. Mac/v7 is not but the patch is straightforward.
Our hope is that the KiCad API project results in a first-party wrapper that users are happy with, but of course we have to prove that out. We have no issue with third-party wrappers existing, of course – the only reason Mark and I are commenting here is so that people know what is coming on the KiCad side and can make informed decisions about what to spend time on.
We currently plan to share details of the new API, including ways people can get involved to work on things like Python wrappers, by the middle of 2024.
I’ve said this before elsewhere, but maybe it bears repeating: my personal opinion is that the current form of SWIG-based Python bindings to KiCad probably never should have been released. They were done in a way that pretty much made it impossible to provide a stable plugin API for Python developers, and even though such stability was never really advertised by the KiCad team, it was (reasonably) expected by plugin authors. On the other hand, despite the pain, they have allowed a wide range of interesting things to be developed without the KiCad team having ever come up with an API. So, I have mixed feelings.
The API we’re working on will be an actual API, with actual stability and compatibility guidelines and expectations.
That said: The new API will never be available to KiCad 8 or earlier. So, projects like this that are aiming to support older KiCad versions still are useful (with the caveat that we don’t really recommend people stay on older KiCad versions)
Yeah, if i’d start from scratch today I’d probably base my plugins on kicad-python. But as I started during V4 nightlies (4.99) I got used to using the API directly. As for the fluid nature of the API, besides being occasional PITA it also forced me to implement some kind of unit/integration tests so that I could catch API changes quickly and automatically. And with the introduction of the PCM where I was having three plugins I wanted to be available, this pushed me to develop an automated plugin packaging procedure so that besides fixing a bug, release of the patched version requires minimal effort. Expanding this to support V6, V7 and V8 was a natural progress
And fluidity of the API was not the major PITA. The following items cause way more grief for me than API changes:
plugin installation procedure before the PCM (where are the folders, …
ubuntu wxpython/wxwidgets mess during the 5.0.x
schematics file format changes (some of my plugins have to parse the schematics files to gather additional info
completely reasonable user feature requests that the code was not architectured to support requiring major refactoring
Again, for me the stability/fluidity of the API was a nuisance bot not really a serious issue. But I know that some had harder time coping with the changes. But having the API resulted in a plethora of different solutions to different problems, and I enjoy looking at this solutions. So from my perspective this is quite worth of a couple of (additional) gray hairs the plugin developers got.
My group does a lot of scripting for CAD. We have written many plugins and an internal common codebase for them. The grand majority don’t get published because the creator or employer a) needs to keep private or b) can’t justify time for public software maintenance. That would take too much time away from hardware development. Perhaps my group and collaborators are the only ones relying on a substantial internal codebase… that I don’t know.
SWIG was the main reason why we migrated from Eagle years ago. We still consider it a differentiating feature of KiCad in the sense that it enables us to fabricate things that would be insane to design in Eagle or EasyEDA or the other ones lacking scripting support. The community aspect is cool but not the thing that meaningfully enhances our engineering capability (the primary exception is replicate layout). The team has been very happy with SWIG. kicad-python is how we continue being happy with SWIG
It is not entirely about people using older versions. People write code for the current version, which becomes legacy code in <12 months. Some legacy code is vital, needs to work next year whether or not the creators want to actively maintain them.
So maybe more accurate: having API that is stable since 2019-ish will be increasingly relevant for users relying on code created before first-class API. This number is unknown and may just be one group+collaborators. Otherwise, I agree, it will not be very relevant for plugins created after first-class API.
Good point. That was a significant downside. SWIG itself is upfront about being unmaintainable, yet exposing what looked like “an API” introduced (a perception of) maintenance liability. I’ve read in this forum and Gitlab that python is not a high priority, and I almost always agree with the reasoning. What seems more reasonable is to externalize liability for python stability to python programmers who want stability. So that is what we did for kicad-python.
Now, the significant features of kicad-python are stodgy software things (API contract, environment management, geometric regression testing). Using syntax that many hardware designers recognize is obviously a feature, but as a sole feature – as was the original idea – syntax is not solving a vital problem. If you add the stodgy stuff, then you can get at this key problem of developing stable plugin and batch code for pcbnew.
Our transistions through v6/v7 (now v7/v8) were also smooth. Very similar experience. I made changes such as this in kicad-python, then all of the group’s and collaborators’ codebases became compatible with v7. The group members could then upgrade to v7 whenever they felt like it. The PCM was a fantastic addition that should accomplish a similar thing.
kigadgets is really just the new kicad-python in a deployable, maintainable form. Since this conception of API worked well for us and continues to work for group members using nightlies, I figured it was due for a proper release, easy install script, issue page, CI testing, that sort of stuff needed to realistically uphold an API contract. Name change is to disambiguate with the syntax-only origins of kicad-python (and to disambiguate with all the projects that match ki*py*). The repo however has remained a fork because all those years of groundwork on kicad-python were necessary.
Sorry I have gone way off topic from IPC/RPC. I had not realized that IPC and scripting might have intertwined fates. Python implementation of IPC would be pretty straightforward, but it sounds like IPC itself is not the sole interest.
.
