I am working on a small python script where I want to utilize functions I defined in a different python module. There I noticed that the python interpreter used by KiCad does not utilizes the systems PYTHONPATH but has its own. Obviously I could modify the path before I do my imports but this normally is not the recommended way.
Does anyone knows a way to modify the PYTHONPATH used by KiCads python interpreter permanently like you would normally do with a .bashrc?
Depending on your method of installation of KiCad (and your version), this may be done automatically.
Could you post up some more details of your system or even better, a Dockerfile that reproduces your condition?
As an example to this, I currently have a Dockerfile that’s installing a particular version of Kicad 7.0.0 through the PPA and apt-get. This then allows me to import pcbnew directly from my root python installation. Dockerfile snippet attached below for your reference.
FROM ubuntu:22.04
RUN apt update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y software-properties-common
RUN add-apt-repository --yes ppa:kicad/kicad-7.0-releases
RUN apt-get update
RUN apt-get install -y kicad=7.0.0-1-202302270924+da2b9df05c~171~ubuntu22.04.1
In addition to this (in case it’s useful to you or anyone from Google): I also run all my scripts through a virtualenv (using pipenv) on my native machine and in order to allow access to this, you either need to install the virtualenv with access to the root site-packages or copy the pcbnew.py and _pcbnew.so files into your virtualenv lib. Again I’ve attached an example below for reference.
PIPENV_VENV_IN_PROJECT="enabled" pipenv install
cp /usr/lib/python3/dist-packages/pcbnew.py .venv/lib/python3.10/site-packages/pcbnew.py
cp /usr/lib/python3/dist-packages/_pcbnew.so .venv/lib/python3.10/site-packages/_pcbnew.so
$ pipenv run python
Python 3.10.8 (main, Oct 12 2022, 19:14:26) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pcbnew
>>> pcbnew.BOARD() # Yay, working.
<pcbnew.BOARD; proxy of <pcbnew.BOARD; proxy of <Swig Object of type 'BOARD *' at 0x7f41c4b32cd0> > >
I don’t think you got my question there or I am completely off, not sure. I have the packages I want all local but the problem is: not all my modules were installed with pip. some of them are just sitting somewhere in my file structure because they are cloned from git. So the import in python wont work unless I modify the PYTHONPATH to include the path to my modules. This you can do in two ways (at least these are the ones I know):
modifying the path with sys.path.append() inside the python script before calling the import.
Adding your custom path globally and persistent to your PYTHONPATH e.g. in the .bashrc
option 1 is considered not the cleanest method and I try to not use it here but I fail to find an equivalent for the .bashrc for kicads python interpreter.
The most recent stable versions of KiCad should install the correct files directly into your python site-packages automatically - you should not need to modify your PYTHONPATH at all. The Dockerfile I posted is an example of a setup where this is true. If you build and run that docker, you can access the pcbnew from a python3 console. Please provide more information on your system, installation method and KiCad version so we can understand why this isn’t the case for you.
No you still don’t get my point. I don’t want to use modules which are part of the kicad install or any pip module. I want to use self defined modules outside of the normal python package path.
Question is, if there is any other way to make the interpreter aware that this package actually exists apart from using sys.path.append('~/workspace/awesome-package/').
EDIT: This reply is incorrect, leaving the original here for clarity of the conversation.
Your KiCad interpreter will be using your system python in most cases (you still haven’t provided any system info so I don’t know if this is true in your case but probably is). So all you need to do is append the package containing your packages like you would with any other addition to the PYTHONPATH.
For your example, add to your ~/.bashrc the following:
Then run source ~/.bashrc to update it and you should be good to go. If you can access it from your python console, KiCad should be able to access it too (again, dependent on your setup).
that is what exactly not works. Calling sys.path() inside KiCad lists only the systems path + kicads own pathes. any export to PYTHONPATH done via .bashrc or also with terminal does not show up there.
Added version info in the first post.
Yep, sorry - I realized that the quick test I ran to verify the above didn’t take effect because I didn’t Refresh Plugins. It does not appear to have access to the PYTHONPATH env var. Give me a bit to poke around - unfortunately I’m running on the PPA installed v7.0.0 which currently has a bug that doesn’t support the scripting console.
Hmm ok, so I’ve dug around everywhere and I can’t actually see an easy way to just directly manipulate an environment variable that the KiCad python will use. A method might exist but we’ll have to wait for someone else to chime in to help on that. It appears that the KiCad packager (sensibly) isolates any system level addition to the PYTHONPATH but weirdly in my case it still uses my system python so it has access to my dist-packages, just not the PYTHONPATH.
Some other suggestions for solving your root problem:
Create a single python file that you put into the scripts dir that appends all your target directories to sys.path manually. This works totally fine and all you’d have to do is import it from your action plugin python files. It’s just annoying to maintain it as a unique definition of the PYTHONPATH. This is my preferred option.
You could symlink the directories you want to access into the scripting directory so that from the perspective of any files there, they are in the same root directory.
You could package up the things you want to import and install them with pip on the python that your KiCad installation uses.
You could run your scripts externally from KiCad like I originally thought you wanted to do. Downside to this is just that you can’t directly interact with things like GUI selections you’ve made.
Yeah quite curios that only the PYTHONPATH is not used here. Maybe it has something to do with kiCad itself adding a lot of paths to the sys.path. Could be that all system wide changes to the PYTHONPATH simply get overwritten. Maybe I will find time later to check this in KiCads source code.
Thanks for the great workaround options there. I think option one comes as close to what I want (single point of maintenance, updates with bash script) without having to much overhead or ugly corner cases.
You are way overcomplicating things and also missing one important piece: gui apps in linux don’t inherit environment vars from bash.
It’s easy to check if kicad messes with PYTHONPATH.
Launch it directly from terminal while passing custom variable:
Check sys.path in scripting console:
Now that you know that KiCad does respect environment variables logical conclusion will be that whatever you write in your .bashrc does not actually have an effect on what variables kicad launches with. And it makes sense because .bashrc is only executed when bash initializes and if you launch kicad from menu shortcut or other gui launch method bash has nothing to do with it.
So how do you set PYTHONPATH so that kicad sees it? It depends on your distribution if it has some method to set env vars for all gui apps. But you can always modify the shortcut you run kicad from or launch directly from terminal.
In hind sight, that’s obviously the way to set the PYTHONPATH env var for KiCad, my bad. But I still think keeping a python file in the scripts dir that sys.appends the desired paths is a slightly cleaner solution than modifying the shortcut because it keeps everything together and easy to maintain.
It would be nice to have a properly isolated environment though (such as a virtualenv) so I could install packages to use in KiCad that weren’t also in my system installation. Is there any easy way to point KiCad to use a separate python bin for the scripting console and plugins?
In current architecture kicad doesn’t use any external python binary, it links to python libs directly and calls python scripts using python api. So you can’t use another interpreter.
You can however still create a virtualenv and install packages there. Just pass it’s site-packages dir through PYTHONPATH to kicad. As long as you are not installing wxpython in that virtualenv or something else kicad itself uses it should work fine.
You can do that too. Just keep in mind that scripts in scripting dir are executed/initialized every time plugins are loaded which is every time pcbnew or fp editor is opened. So make sure to not duplicate path records.
This is good advice, only downside is that you’re forced to use your system root site-packages which might conflict with what you want to setup in the virtualenv. But this isn’t important enough to warrant any thought until someone actually needs this.
Hmm, I guess the only complication this introduces is that you’ll presumably sometimes need to have the path appended before executing some of the other action plugins so you’ll still need to import it in those plugins. That means you need to conditionally add the paths only if they don’t exist. That way it doesn’t matter if the script gets run multiple times from several plugins that import it or directly by KiCad on startup. Not a difficult solution though.
So I missed the obvious so I will write it down here in case others are not aware. This only works on debian-based systems including ubuntu:
Modifying .bashrc and also .profile will not work as KiCad does not launch a shell (here was my error assuming kicad would do that). Instead you have to modify (or create) the .pam_environment in your home folder and add lines containing the path to your site-packages similar like you would do in a .bashrc:
Does this work? If so you might want to be careful with it because it will now be setting your PYTHONPATH for several other applications which seems like a risky move.
To be honest, I didn’t really know about pam_env until this prompted me to look it up.
It does and yes if you use it you should be careful not to override any functions/classes set by other internal modules. But you can also simply put your path to the end of the path so your modules have the least priority:
