Net-aware pcb diff tool (kicad-python demo)

[pointhi]

To further develop kicad-python as proposed in (Building a proposal for an (official) high level python API) I implemented a net-aware diff tool. This means it is not just a simple graphic differ of the copper plane as many other existing ones, but shows which nets changed in what way (Based on their name). Furthermore, it is based on KiCad internal polygon operators instead of some 3rd-party applications which shows there is not much work required to get such tool into the core. (https://bugs.launchpad.net/kicad/+bug/1774887)

At the moment, the diff is outputted using mathplotlib, but this is quite easy to be change. Actually, when someone manages to get a GAL canvas running in python we wouldn’t even need to convert it.

The tool is not finished although. Currently no draw segments are supported, as well as it seems the clearances have to be imported from somewhere to be actually an accurate representation of the polygon.

pointhi/kicad-python/blob/master/examples/pcbnew_diff/pcbnew_diff.py

[pmx]

Hello,

how to test-try this ? Or more specifically, how to get the Kicad python libs ? I just didn’t find the info - I’m probably dumb

I’m running under Windows, if that matters.

[pointhi]

I only use Linux, but the script should be working on windows as well. The commands could probably differ a bit to the Linux workflow:

1. Install the dependencies kicad-python and matplotlib

   pip2 install kicad-python matplotlib
   

2. Run the posted script

   ./pcbnew_diff.py old_board.kicad_pcb new_board.kicad_pcb --layer=F.Cu
   

[Rerouter]

I just went and linked that bug report to the blueprint discussing such a tool.

[pmx]

Something went wrong…

    Traceback (most recent call last):
      File "C:\Users\pmx\Documents\Kicad test diff\kicad-python\examples\pcbnew_diff\pcbnew_diff.py", line 21, in <module>
        from kicad.pcbnew import Board, Layer
      File "C:\Python27\lib\site-packages\kicad\pcbnew\__init__.py", line 16, in <module>
        from kicad.pcbnew.board import Board                # noqa: F401
      File "C:\Python27\lib\site-packages\kicad\pcbnew\board.py", line 16, in <module>
        from kicad.pcbnew.boarditem import BoardItem, from_board_item
      File "C:\Python27\lib\site-packages\kicad\pcbnew\boarditem.py", line 16, in <module>
        from kicad.pcbnew.layer import Layer, LayerSet
      File "C:\Python27\lib\site-packages\kicad\pcbnew\layer.py", line 18, in <module>
        from kicad._native import _pcbnew
      File "C:\Python27\lib\site-packages\kicad\_native.py", line 22, in <module>
        raise e
    **ImportError: No module named pcbnew**

From _native.py :

import sys

try:
_pcbnew = __ import __(‘pcbnew’) # We need to import the pcbnew module this way
except ImportError as e:
[…]

I don’t really understand the reason of this construction (my python skills are quite basic).
BTW, I can’t find a pcbnew module anywhere.

FYI : Python 2.7 here

Any clue ?
Thanks in advance !

[pointhi]

@pmx, are you able to “import pcbnew” using the shell? You need to have KiCad installed for this, and probably you need to use the python shell located in the KiCad directory to find the package on Windows.

[Rene_Poschl]

Is there a requirement on the kicad version?

[pointhi]

KiCad 5 should work. CI tests running on current nightly and I’m also running nightly. If someone finds issues with 4.0.7, please report, so I can get workarounds into it if possible.

[pmx]

Yes, I tested “import pcbnew” in a std python shell.
But not in a kicad shell (didn’t know this shell existed).

“import pcbnew” works in the Kicad shell. I gess it means that pcbnew.py is part of the Kicad program install.
Shouldn’t it be also part of the “pip install kicad” process ?

I dropped pcbnew.py into my main Ptython site-packages dir, the error is now :

from kicad._native import _pcbnew

File “C:\Python27\lib\site-packages\kicad_native.py”, line 22, in
raise e
ImportError: DLL load failed: %1 is not a valid Win32 application.

32 bits vs 64 bits pb ? (32 bits python, 64 bits Kicad ?)

(I have to use 32 bits Python 2.7. on this machine. )

I will try a to install 32bits Kicad.

Too bad, I have to downlload the whole thing,1.0Gb and this includes the libs (on windows). I already have these hudge libs…

The STEP models SHOULD be compress in the installer, and even better make a separate download for the libs! If the windows packager reads me…

Maybe I will reinstall a Linux partition, lost it after my last HDD crash (feel like I’ve done that a million times !).

[pointhi]

In “C://Program Files/KiCad” (I think this is the correct path) there is also a Python.exe which can be used. I thought about this one when proposed the “Kicad delivered Python Shell”. I haven’t tested it thoughtfully, but kicad-python could have a name clash with some KiCad internals in the internal python gui

Nevertheless, I don’t really use Windows productive, especially not for programming.

[pmx]

Finally installed the 32bits version of Kicad 5.0. Same result.

Tried to launch the python interpreter supplied with Kicad.
No better result.

I will suspend the tests for now (until I reinstall Linux).

[bobc]

@pmx

I dropped pcbnew.py into my main Ptython site-packages dir, the error is now :

from kicad._native import _pcbnew

File “C:\Python27\lib\site-packages\kicad_native.py”, line 22, in
raise e
ImportError: DLL load failed: %1 is not a valid Win32 application.

32 bits vs 64 bits pb ? (32 bits python, 64 bits Kicad ?)

I get the same error (Win7, 64). No one has come up with a solution, but plenty of people say “it should work” but without having tried it and the fact is it doesn’t work.

@pointhi

In “C://Program Files/KiCad” (I think this is the correct path) there is also a Python.exe which can be used. I thought about this one when proposed the “Kicad delivered Python Shell”.

On Windows this is the only way to successfully import pcbnew. The KiCad python installation is a bit archaic which leads to problems installing packages into it but there is no alternative.

[maui]

On windows the KiCAD pip2 doesn’t allow to install mathplotlib and the lib is not bundled with the installer
… so the script will not run…

[pmx]

OK, I think I understand better the Windows + Python + Kicad problem.

First, when using Kicad’s private Python shell, you have to force the installation of the dependencies (like matplotlib) in Kicad’s private Python dir. By default, pip2 installs the modules into the system’s Python2.7 directory… Obvious, once you think about it ! The command line is described later.

Second point, and the most problematic, I finally discovered that the Python shell from the Windows Kicad distro is a very special beast, built with GCC.

The essential file _pcbnew.pyd , part of the Kicad Python distribution (more or less a windows DLL in disguise) is built with GCC, too. And probably Kicad itself.

The official Python shell and modules are build with Microsoft tools (VC++).

Tracing the errors confirms the binary incompatibility (ABI : Application Binary Interface).

Then, any official python module (VC++) that tries to load some binary .pyd files will not work in the Kicad python shell (GCC). And vice-versa.

Specifically : Kicad python relies on the file _pcbnew.pyd (compiled with GCC).

The plugin we are testing (pcbnew_diff.py) relies on matplotlib. matplotlib uses numpy. And numpy tries to load the multiarray module : multiarray.pyd (VC++ Binary file).

If i use the Kicad’s python shell, it fails on multiarray.pyd.
If I use the official python shell, it fails on _pcbnew.pyd.

Indeed, this doesn’t happen on Linux, where GCC rules, and where developers have only one main Python installation.

Bottom line : based on this first experience, I can safely predict that Python extensions/plugins, that rely directly or indirectly on .pyd files (like matplotlib), are guaranteed to fail on Windows, unless developers :

• provide specially compiled version of these Python modules. Modules have to be installed in Kicad’s private Python directory. The extension/plugin will only work within the Kicad Python installation.

OR

• switch the Windows port to Visual C++ (may be challenging!!). In this case, a private Python is not necessary (I guess…), .

• If a private Python installation is still required for some reasons, installation of external modules will have to target the Kicad install dir :
  pip2 install PACKAGE --target DIR --upgrade (tested it).
  In my case, DIR was “C:\Program Files (x86)\KiCad\lib\python2.7\site-packages”

Comments are welcome.

My 2 python-cents !

[pmx]

Two more thoughts :

I didn’t explore 32bits / 64bits compatibility (enough for today !)

Among the environment variables set by the windows installer, I didn’t find the top Kicad install dir (to redirect pip2 install).

[PCB_Wiz]

@pmx

Bottom line : based on this first experience, I can safely predict that Python extensions/plugins, that rely directly or indirectly on .pyd files (like matplotlib), are guaranteed to fail on Windows, unless developers :

• provide specially compiled version of these Python modules. Modules have to be installed in Kicad’s private Python directory. The extension/plugin will only work within the Kicad Python installation.

Hmm… sounding messy.

The OP actually has another impressive Python effort here, around Python 3 support in kiCad

Moving from Python to Lisp/Lua

I think improving our current python implementation is the way to go. I started with an abstraction layer as already pointed out (prototype: https://github.com/pointhi/kicad-python). It already works, but I think it would be more important at the moment to get the wxPython implementation to Phoenix as well as adding support for Python 3. If anyone has the interest to improve the current implementation, I’m happy to see contributors. I think it would be even a good idea to drop Python 2 support …

That thread probably even needs a moderator split of that Topic, as V3 Python, is not really lisp/lua

I wonder how V3 python could/might resolve these windows build issues ?

[pmx]

Please, not Lisp…

Everybody should knows what LISP really means : Lost In a Sea of Parenthesis .

Years ago, I tried very hard to do Emacs lisp programming.
I didn’t succeed.
Then, I tried very hard to read Emacs lisp programs.
Another fail, witch was the definitive proof that I’m a perfectly normal programmer.

[PCB_Wiz]

@pmx

Please, not Lisp…

Relax … You should read the link, and the thread-fork, which covers work in getting Python 3 working.

[pmx]

I read the link, I was just joking, indeed…

(defun read_lisp ()
  if (> parenthesis 99)
    let (programmer )
)