Original question: I have a question regarding PCM - what about plugins that have third-party dependencies (i.e., Python libraries). Does the installation process via PCM takes this into account? Currently, it doesn’t look so. Is it expected that the user will have to install such dependencies manually?
@Jonathan_Haas: Technically yes, but not in a simple (and maintainable) manner. The problem is with binary dependencies (e.g., numpy) that are platform-dependent. If I try to include it, it makes my life extremely hard and the plugin prone to bugs. Why duplicate something Pip can already handle well?
For now there is no way to indicate pip dependencies, this is planned for the future.
Best course of action is to avoid using such dependencies. In most cases you don’t really need numpy just to use a couple features from it.
Numpy was just an example of a well-known library that has a binary dependency. E.g. KiKit cannot work without Shapely which also presents a binary dependency.
Just out of curiosity; why didn’t you go by specifying that plugins are python packages that follow the standard conventions (and thus, feature setup.py that can manage the dependencies)?
Because that’s not how KiCad’s plugin loading system works.
Also having the package management be done entirely in cpp and not touch anything python was one of the requirements of the feature design.
It’s up to plugin author to implement that popup. KiCad just tries to import the plugin package/module, if it fails it’s ignored.
You can put your import into try/catch block and fail gracefully when plugin is invoked.
It’s relatively trivial to find out what is missing, if you know your way around python. Likewise for Linux users a simple “emerge numpy”. (or whatever you package manager provides) is just as trivial.
However, this is Linux and kicad uses the system python. For Windows it’s within its working directory and thus a bit more effort is involved - at least Windows build has dropped py27, you cannot install numpy with kicad with 5.1.x …
This is a hard problem to solve. As it stands currently, there is not an easy way to have plugins pull in other Python packages as dependencies in a cross-platform way that is reliable. We are not going to implement a solution that only works on Linux even if that’s the easy way.
We are still thinking about and discussing ways to improve this in the future. The most ideal way would be if we can expose a working pip setup somehow on Windows/Mac, but it’s a big task.
Whatever we do, it’s going to need to work on an isolated, non-elevated Python installation. So, installing packages that require compilation on the host or installation system-wide is never going to be possible from within KiCad.
For now, the best advice I have is to reduce dependencies on third-party libraries that you can’t bring in-tree as source dependencies. If this is not possible, design your plugin so that it fails gracefully if a dependency is not present and give your users instructions on how to install it.
Because we are using MSVC now, most python libraries have precompiled binaries in pypi that are compatible…in theory so no building per PC required like Linux land and it’s ABI hell that requires it. There are some limited scenarios the precompiled might not work but that’s because the python ecosystem is not designed for embedded python.
I can make pip on Windows work 70% of the way there (I should stop being lazy and drop in the one file to do it). The problem is, it was not designed for embedded python. It hardcodes the target paths to share with system installed python which we do not want as that will absolutely cause conflicts.
For the record, I intentionally bundle numpy with the Windows kicad.
That’s fine! To those reading this thread, if we are to have some kind of better dependency situation for Python plugins, we need volunteer(s) with experience in C++ and Python and with Windows/Visual Studio/vcpkg to move this forward. If we can get a proof of concept working in Windows, the rest of the team can help with porting it to macOS. If anyone is interested in this project, please reach out on the developers’ mailing list.
This isn’t a vcpkg Windows/vcpkg/Visual Studio problem.
It’s this bit that is the problem. There is no way to override these. Yes there is a pip.ini file where you can try and override paths a little bit, and it works maybe 25% of the time, it’ll even print it loaded the paths from pip.ini…and then proceed to not use them at all.
The problem of patching pip is pips constant hassling to update it to match pypi. At which you do it because the pypi changed their api or signing key or whatever and suddenly you have users with broken pip again that will run the auto update. It needs to be a upstream fix.
I’m trying to get more familiar with pip. I don’t understand yet what is actually needed and wanted, and what marekr has tried. Are we talking about activating pip from KiCad, or are we talking about just getting pip to work for Windows users so that they could themselves follow some instructions to install pip packages manually? Have you tried or considered different possibilities (which might or might not help) which are given in https://packaging.python.org/tutorials/installing-packages/ ? Virtual environments?
And what are the cases which must be taken care of, for example what should I install and do to trigger problems? Do you have some example plugin which tries to use a pip package which doesn’t work (one of those nonworking 30% cases)?
Either would be possible, but the blocker to either approach is to make pip work from the packaged Python in Windows (and macOS). Unfortunately as far as we can tell, the only path here is to patch pip to accept non-hard-coded paths, without using virtual environments or anything else like that.
Once that is accomplished, details like “can you run pip from inside KiCad, or do you have to open a terminal window” are smaller issues.
Just depend on any Python package that isn’t in the standard Python distribution and we don’t already ship with Windows (numpy, pillow, and six, I think)
You don’t even have to install anything: say you want to be able to use scipy. You should be able to open the KiCad Python console and type import scipy and have it work. Of course it won’t work until you find a way to have pip install prebuilt binaries of scipy into the right location.
Sigh, well I already know a plethora of solutions that’ll fix this without patching pip.
What I like right now. Create a “Launch kicad pip console” in the windows start menu and maybe elsewhere. This console will just be a bat file that sets all the environment variables to make pip use all the correct kicad directories, I can even shove a pip.ini down it’s throat by force at that point.
The question is, where will pip be told to installed packages? AppData, Documents/kicad//scripting? PRogramData (all users), etc, etc
