Python is the defacto scripting language for many application suites. It’s simply not sensible to attempt to cater to every pet programming language that people wish to use - it fragments resources and creates too many moving targets in an already complex application suite.
I’d prefer solid support of one language through all of KiCad (especially with a good interface to eeschema AND pcbnew) rather than dodgy support of a few languages scratching random itches. Python is a great way to do this (readibility, ease of use, existing libraries, etc) and I’m glad that the project is continuing in that direction.
If there are several dozen script writers, then the abstraction layer would create a stable data model for them. Then any changes in the underlying KiCad data model would be hidden from them. If the KiCad data model changed, then whoever is in charge of the abstraction layer can fix the incompatibilities for every script in one place without having to know how each individual script works.
The alternative is to make the script writers responsible, and in a lot of cases they’re not maintaining their scripts even if others have come to depend on them. That makes scripting less valuable for the KiCad community because they’ll never know if a given script will run or not. And scripts should be a valuable feature for the community because they bring new features without the delays and complexity of modifying the KiCad core. (Certainly Eagle has benefited from a large collection of scripts; they just chose to implement them with an ad-hoc, closed language which caused its own set of problems.)
Or the python api has testcases that check that nothing did break and every change to the underlying model must go through these checks. (Only changes that do not break the testcases are allowed in. Means everyone contributing to kicad is then responsible for keeping the api alive.)
I would also like to see something similar to file format related things. (right now different versions of kicad write files slightly differently. Some put in additional whitespace chars, some order lines differently, …) This just makes live hard for everyone who uses version control. (I think i already proposed this but i don’t think the devs take it very seriously. So i resort to simply bugging them with every change they make.)
I would also suggest that a reasonable set of libraries be supported as well… if scripts pull in external libraries you can be sure they will break sooner or later.
It would be nice if scripts like KiCost just worked out of the box in the KiCad maintained python enviornment…
I don’t know if that’s ever going to be practical. Something like KiCost uses libraries for things like web scraping and spreadsheet manipulation libraries. KiCad can’t drag the entire Python ecosystem around with it.
What would be nice would be to substitute an external Python interpreter and set of libraries for the internal ones used by KiCad. Then you could load the libraries you want. Maybe that’s possible now, I don’t know. If not, I don’t know all the changes that would be necessary to make it happen.
I think Scheme would be better than lisp in general. Mostly they’re compatible, it’s just scheme doesn’t have cons so doesn’t need to implement cons cells and so can have more efficient internal representation than lisp. Lists for example can be represented as arrays with no need for cdr’s.
Scheme (and lisp) has some big advantages over python. The most obvious is the ability to programmatically construct code, such as data filters or sort ordering functions based on complex criteria. For example, for footprint or component library searches (# of pins, substring, specific library or substring, patterns, etc) - where instead of a monolithic comparison function that takea a gazillion parameters, a function is created that constructs a comparison function based on search criteria, and then this is used by a generic search. It’s also easier to verify since the filter or comparison constructor function can be tested in isolation - feed it inputs and see what code it outputs, without actually have to unit test against a giant test data set.
Of course, the ecosystem is much better for python and there are libraries for virtually any task.
You are probably right, it would be nice if there could be a maintained set of libraries that large that just worked and allowed use to just run tools like KiCost but that’s a huge maintenance burden. Honestly I think python is overkill for what it sounds like people are wanting to do with it.
Scheme/Lisp is a bit too esoteric for most… I’ve never written too much serious in it other than a sudoku solver.
Lua really is the go to language for simple scripting though, thats why it was developed to being with… and there is an updated wxLua bindings library to go along with it. https://github.com/pkulchenko/wxlua
And since Lua is designed to interface with C/C++ you don’t end up with the whole problem of having a secondary wrapper for the swig based API…
Exactly - change of top scripting language, does not solve the underlying problem, but creates many more…
Better to focus on fixing the areas of weakness, whilst keeping what works already.
There is a lot of Python out there already for kicad
I also see a growing number of Python Embedded downloaders, as Python moves to become more of a workbench standard.
Getting back to @cb88’s original point about fragile Python environments, this comic captures the situation well:
Depending on your system, you might have a system package manager (apt-get or homebrew) that installs Python and various versions of Python libraries, and then Python based tools like pip that will install possibly conflicting versions system-wide. It can become a mess when you start using both to upgrade library versions.
And then there are “virtual environments” and virtual environment wrappers that alleviate some of the problem, but they require work to setup. Too much, in my opinion, but I don’t particularly enjoy dependency management, even if it is unavoidable.
The best way to manage paths and dependencies I’ve found so far is with pipenv and it’s Pipfiles. pipenv makes it pretty easy to track the dependencies for an individual Python program and automatically install them into a self-contained virtual environment. The best part is that the Pipfile is part of the source repository maintained by the developer, and the user just needs to run pipenv shell to have the same build and run-time environment as the developer.
Perhaps providing Pipfiles with scripts and/or the KiCAD Python API would be helpful? I haven’t used the scripting API at all, so I don’t know.
All that said, I agree that Python is a fine choice for the KiCAD scripting language. I just wanted to acknowledge the original point that dependency hell is no fun. You just can’t avoid it.
I think improving our current python implementation is the way to go. I started with an abstraction layer as already pointed out (prototype: https://github.com/pointhi/kicad-python). It already works, but I think it would be more important at the moment to get the wxPython implementation to Phoenix as well as adding support for Python 3. If anyone has the interest to improve the current implementation, I’m happy to see contributors.
I think it would be even a good idea to drop Python 2 support completely if possible after getting Python 3, or at least output a big warning when someone tries to use it (Doesn’t make sense to write scripts for it: https://pythonclock.org).
pipenv looks interesting. Don’t know how this cooperates when you have multiple plugins with different virtual environments in one application.
Given the time it takes for development, there seems to be a fair amount of catch-up that occurs as the codebase is shifting to more recent libraries. I think it would save time and energy to be more brutal in dropping support for 2.7 and going straight to 3.
That’s easy to say, if you ignore the large and widespread user established code base.
Reality is, there is a lot of proven 2,7 Python out there, and that is not possible/practical to port overnight.
Best path is to encourage the dual-version support as described in the link above, so that over time V3 can be used more and more.
Some companies may choose to install 2 Kicads, a Py2.7 for legacy script use, and a Py3 for new development.
I’m very aware that there is a large base of support and libraries that use 2.7. I just believe that best path forward with limited resources is to look to the future rather than splitting attention between 2.7 and 3. It’s very much a chicken and egg problem, as we hold onto 2.7 because there is not enough support for 3, but this continues to degrade the usage of 3 as everyone is afraid to let go of 2.7.
Python support will not get better while trying to satisfy both branches. If it comes to committing to a path, I say look to the future. Sometimes things need to break to get better.
“large base of support and libraries that use 2.7”
Which are irrelevant to KiCad itself as it isn’t going to ship with a lot of external libraries such as would be needed to internally support large plugins like KiCost etc… right, just enough to allow it to act as an extension language for Kicad?
It’s not irrelevant to those of us who like being able to use external libraries. For example I am trying to automate many processes using KiCad using dxf placement, optimisation of component location placement and autogeneration of schematics, netlists and pcbs. Hence external library support is important - but I would also like it stable so it’s not going to break with every release.
I do understand this is a lot of work and a long way off.
It’s intended for commercial purposes, but I plan on releasing a more genericised version when I have it more robust and less held together by bandaids and swearing
