Issue with importing pcbnew using pytest in a virtualenv

Problem Summary
importing pcbnew in a python script from within an Ubuntu 22.04 based docker image works but the same import fails with the error below when run from pytest in the same environment. I’m not sure if the issue lies in how I’ve linked in the kicad swig interface module or is something inherent with pytest or something else.

kicad_fplib_tools/utils.py:15: in <module>
    import pcbnew
/usr/lib/kicad/lib/python3/dist-packages/pcbnew.py:15: in <module>
    import _pcbnew
E   ImportError: /lib/x86_64-linux-gnu/libfreeimage.so.3: undefined symbol: _TIFFDataSize, version LIBTIFF_4.0

More Details
I’m trying to update a bunch of scripts I developed for KiCad 5.x to work with v7.0.0 and in the process I’m setting up a Dockerfile to automate testing. I’ve setup got an Ubuntu 22.04 Docker image that installs KiCad 7.0.0 using apt and downloads + unpacks a FreeCAD AppImage. I run all my local tests on this library and others using pipenv so I’m setting things up to work directly through that.

I currently have the pcbnew module able to be imported after installation by adding the 2 lines below to the Dockerfile after installing KiCad 7.0.0. My previous installations of KiCad added the Swig interface to my pythonpath automatically and this step wasn’t required. If anyone has advice on this too, it would be greatly appreciated - I may create a new topic for this soon too.

ENV PYTHONPATH=${PYTHONPATH}:/usr/lib/kicad/lib/python3/dist-packages
ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib/kicad/lib/x86_64-linux-gnu

When running python within a container using the image, I am able to import the ‘pcbnew’ module without any issues. However when I attempt the import from within pytest, I get the error message quoted above in Problem Summary.

For further clarity, this is a copy paste from the import working correctly from a shell in the container using python directly.

$ docker run --rm -it --entrypoint bash freecad_kicad_tools
root@23e5e687da57:/project_dir#  pipenv run python
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pcbnew
>>> pcbnew.F_Cu
0

And now trying the same while running pytest.

$ docker run --rm -it --entrypoint bash freecad_kicad_toolsroot@e575c4c552c3:/project_dir# pipenv run pytest
===================================================== test session starts ======================================================
platform linux -- Python 3.10.6, pytest-7.2.1, pluggy-1.0.0
rootdir: /project_dir, configfile: setup.cfg, testpaths: tests/
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /project_dir/tests/unit/test_shape_generators.py(16)<module>()
-> from kicad_fplib_tools.utils import KiCadLayer
(Pdb) import pcbnew
*** ImportError: /lib/x86_64-linux-gnu/libfreeimage.so.3: undefined symbol: _TIFFDataSize, version LIBTIFF_4.0

My Environment

  • kicad-cli version > 7.0.0
  • pipenv run python --version > Python 3.10.6
  • pipenv run pytest --version > pytest 7.2.1
  • OS: Ubuntu 22.04.1 LTS (within a docker)

In case this is useful: I originally suspected it to be an issue with pytest manipulating the pythonpath in order to make test files importable. However I also compared the sys.path in both instances, found the paths missing in the vanilla python execution and then manually added them before trying the import again - this did not change the results (vanilla python still imported without issue).

The commands I used to do this and outputs are outlined below if you want to see what I mean specifically.

$ docker run --rm -it --entrypoint bash freecad_kicad_tools
root@abe1e66402c2:/project_dir# pipenv run pytest
===================================================== test session starts ======================================================
platform linux -- Python 3.10.6, pytest-7.2.1, pluggy-1.0.0
rootdir: /project_dir, configfile: setup.cfg, testpaths: tests/
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /project_dir/tests/unit/test_shape_generators.py(16)<module>()
-> from kicad_fplib_tools.utils import KiCadLayer
(Pdb) import sys
(Pdb) from pprint import pprint
(Pdb) pprint(sys.path)
['/project_dir/tests/unit',
 '/project_dir/tests',
 '/app/freecad/squashfs-root/usr/lib/python3.10/site-packages/git/ext/gitdb',
 '/project_dir/.venv/bin',
 '/project_dir',
 '/usr/lib/kicad/lib/python3/dist-packages',
 '/app/freecad/squashfs-root/usr/lib',
 '/app/freecad/squashfs-root/usr/lib/python3.10/site-packages',
 '/usr/lib/python310.zip',
 '/usr/lib/python3.10',
 '/usr/lib/python3.10/lib-dynload',
 '/project_dir/.venv/lib/python3.10/site-packages']
$ docker run --rm -it --entrypoint bash freecad_kicad_tools
root@8389cc64abc9:/project_dir# pipenv run python
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> from pprint import pprint
>>> pprint(sys.path)
['',
 '/project_dir',
 '/usr/lib/kicad/lib/python3/dist-packages',
 '/app/freecad/squashfs-root/usr/lib',
 '/app/freecad/squashfs-root/usr/lib/python3.10/site-packages',
 '/usr/lib/python310.zip',
 '/usr/lib/python3.10',
 '/usr/lib/python3.10/lib-dynload',
 '/project_dir/.venv/lib/python3.10/site-packages']
>>> sys.path += ['/project_dir/tests/unit', '/project_dir/tests', '/app/freecad/squashfs-root/usr/lib/python3.10/site-packages/git/ext/gitdb', '/project_dir/.venv/bin']
>>> import pcbnew
>>> pcbnew.F_Cu
0

Error you are getting suggests that pytest runs in an environment that is trying to read wrong libfreeimage.so.

Try skipping pipenv and run python3 -m pytest directly (provide full path to python3 from the venv if necessary).

Another thing to check: make sure you use the stable testing ppa instead of release ppa in your image. 7.0.0 had some issues with python packaging in release ppa. Ubuntu 7.0 testing builds

Thank you for the response. I agree with your suggestion that it seems to be reading the incorrect libfreeimage.so

Trying your first suggested workaround yields the same results as before (shell output below).

I am also using what is listed in Install on Linux | KiCad EDA as the stable release PPA for Ubuntu (ppa:kicad/kicad-7.0-releases), I’ve included the first portion of the Dockerfile that installs KiCad below as well for reference.

The next thing I’m going to verify is what the results look like if I remove FreeCAD from the Docker image. Though this wouldn’t be a workaround for me because I do want to be able to use them together in the image. Will update results.

pytest invocation with virtualenv python

$ docker run --rm -it --entrypoint bash freecad_kicad_tools
root@8c286556bf45:/project_dir# .venv/bin/python -m pytest
===================================================== test session starts ======================================================
platform linux -- Python 3.10.6, pytest-7.2.1, pluggy-1.0.0
rootdir: /project_dir, configfile: setup.cfg, testpaths: tests/
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /project_dir/tests/unit/test_shape_generators.py(16)<module>()
-> from kicad_fplib_tools.utils import KiCadLayer
(Pdb) import pcbnew
*** ImportError: /lib/x86_64-linux-gnu/libfreeimage.so.3: undefined symbol: _TIFFDataSize, version LIBTIFF_4.0

Dockerfile with KiCad ppa

FROM ubuntu:latest

# RUN apt-get update
RUN apt update
RUN apt install python3 -y
RUN apt install pip -y
RUN apt install wget -y

########################################################################################
### INSTALLING KICAD
# Necessary for having access to add-apt-repository for KiCad
RUN DEBIAN_FRONTEND=noninteractive apt install software-properties-common -y

# Adding the repository for KiCad 7.0 stable release
RUN add-apt-repository --yes ppa:kicad/kicad-7.0-releases

# Changed the keyserver address according to this forum thread advice.
# https://groups.google.com/g/Kurento/c/Q0KE1b8ol_c?pli=1
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 245D5502FAD7A805
# RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 245D5502FAD7A805

# Install KiCad 7.0
RUN apt update
RUN apt install kicad -y

# These changes weren't needed to be manually performed for KiCad 5.1 - not sure why
# they're needed now.
ENV PYTHONPATH=${PYTHONPATH}:/usr/lib/kicad/lib/python3/dist-packages
ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/lib/kicad/lib/x86_64-linux-gnu
########################################################################################

Because

You really should try the testing ppa. It has 7.0.1 precursor builds, only bugfixes on top of 7.0.0.

You can also try modifying official kicad docker image KiCad / kicad-ci / KiCad CLI Docker · GitLab
and adding any tools you need on top.

Ok so I had everything cached so removing FreeCAD was actually much faster than I expected. This actually allowed pcbnew to be imported from within pytest.

So my remaining hanging issues are:

  • Why does the referenced libfreeimage.so change between ‘vanilla python’ and pytest?
  • How can I setup side by side installations of FreeCAD 0.20.0 and KiCad 7.0.0 so that I can import FreeCAD and pcbnew together? (I’ll look through the forum for similar questions to this).

By the ‘stable testing ppa’ do you mean the nightly ppa?

Thank you for the link for the official KiCad CLI Docker - I will definitely try modifying from that.

More specifically: do you mean this ppa?
ppa:kicad/kicad-7.0-nightly

It has nightly in the name but it’s actually stable testing. It builds from stable 7.0 branch that has only bugfixes. It’s not the same as “nightly ppa” that is development nightly for version 7.99 (unstable, new features and bugs pouring in at a high rate at this moment in time).

Yes

Keep in mind that docker image is for development nightly. There is no official stable docker image yet. You should be able to modify it for v7 by specifying 7.0 branch in checkout here

Understood - thank you for the clarification. I’ll spend today poking around and will post up my results here in <24 hours.

Thanks for your help qu1ck!

Well I found the source of my real issue but not why it occurred in pytest only.

Because I’m pretty jankily using an expanded AppImage for FreeCAD, I’m just extracting it and then linking in a library directory to PYTHONPATH. For some reason, in the original Dockerfile I made for FreeCAD I also decided to link in the python3.10/site-packages directory to PYTHONPATH - I honestly don’t remember why I did that. Removing this from PYTHONPATH didn’t cause any failures in the tests for my freecad scripts and allowed me to import pcbnew from pytest in the freecad + kicad docker.

I’m very aware that there are probably some double-ups and potential conflicts that will arise from using the AppImage together with the ppa installed kicad package. But it’s a simple solution for me at the moment that works for what I need it to. I will eventually go back to trying to start from the official kicad-cli Docker image but I’ll abandon that plan for more pressing issues on my end for now.

In case anyone is interested in my janky use of FreeCAD in an image, the relevant part of the Dockerfile is listed below (after removing the site-packages dir from the PYTHONPATH add-on).

########################################################################################
### INSTALLING FREECAD
RUN mkdir -p /app/freecad
WORKDIR /app/freecad
# This downloads the 0.20.2 FreeCAD AppImage
RUN wget --progress=dot:giga https://github.com/FreeCAD/FreeCAD/releases/download/0.20.2/FreeCAD_0.20.2-2022-12-27-conda-Linux-x86_64-py310.AppImage
# This extracts the AppImage into a sub-folder called 'squashfs-root' then deletes the 
# original AppImage.
RUN chmod +x ./FreeCAD_*.AppImage
RUN ./FreeCAD_*.AppImage --appimage-extract > /dev/null 2>&1
RUN rm FreeCAD_*.AppImage
WORKDIR /
ENV PYTHONPATH=${PYTHONPATH}:/app/freecad/squashfs-root/usr/lib/
########################################################################################

Thank you again qu1ck for your quick responses and direction. I hope to give back to the community here over time.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.