The libraries are hopefully going to be disconnected from the actual software builds, and something that users can download separately.
As to when the data is available - we are in the process of transferring library data to the new repositories. As we do so, we are marking the old libraries to indicate that they will receive no further updates. However the current data will stay in place too.
That is true to an extent, more that some libraries are more “regular” and therefore contain more redundancy.
An extreme example is Pin_Headers.3dshapes, 560MB down to 23MB.
With the developer power available I don’t think it makes sense to start developing library manager. I don’t think anyone can really estimate required effort.
Besides I think that Git and Github are quite an amazing solution as a library database. The only downside I see and it was already mentioned is file oriented setup which limits speed when you have a lot of footprints. This could be improved if footprints would be stored in one file per library as currently the symbols are. So for V6 I would seriously consider if switching symbol library layout one file per symbol is warranted. Otherwise Git and Github already cover most of the use cases. This will be even more obvious with sane repository layout in V5. Additional benefits, which come with using git and have not been widely advertised are:
pull request feature. This is a really great feature which I suspect that library maintainers are quite fond of. I don’t think the integration of new parts into the official library with proper inspection could be done any easier.
The ability to merge personal library with official library. If you clone the official library, make a local branch from master and switch to it, you will still be able to integrate any recent changes to master made by other contributors into your own branch. This is phenomenal property which really has not been widely emphasized (I suspect it will become more obvious with V5 repository layout). Doing it this way makes it much easier to use personalized libraries along the official ones. For example I don’t really like having footprint name visible in Fab layer. I clone the official library, send a script through library which makes footprint names non-visible and commit changes to my local branch. If any changes happen to existing footprints anywhere else I can (or better git can) merge this changes automatically.
Option to have multiple remote repositories. With git you can have multiple remote repositories. The official Kicad supported by Kicad library maintainers and an unofficial one, which could be open to anyone where people would be able to throw anything (as was already expressed in https://forum.kicad.info/t/ughh-the-current-libraries-are-a-mess/8026/29). Then it is up to individual user to choose from which repository he will merge to his local library. You can have additional repository which is available only within the department/company. And you have the same pull request feature to keep the department/company library under tight control (we are currently using this setup). We just need to show how this is done.
As for the size constraints I would package only symbols and footprints with the installer and keep 3D model available as a separate installer.
If the installers would have an option to select where to put the libraries and set the environment variables accordingly it would be even better.
What is really required is good documentation and a YouTube video or two to showcase how to:
use offline libraries without any git integration (I think this has been mostly covered), and it would be quite simple to show with help of installers.
simple git integration via one Git GUI tool (I’d recommend git-gui which is already a part of Git suite and is available for all platforms). This would show git clone and git pull to refresh the repository and maybe even how to switch to older version.
advanced git integration showing advantages of local branches, merges from master and different remote repositories.
@MitjaN what a great synopsis. As I am so invested in the libraries it is often hard to take a step back and develop a good summary, as you have done here.
You’re quite right, the current PR solution is pretty darn good. Especially now that we have checker scripts which do most of the repetitive work for us
This needs to be advertised more, once you have it setup correctly it’s a very powerful feature.
Users are of course free to source libs from wherever they please, or create their own
The footprint and symbol libraries once compressed are very small.
The entire footprint library set is 24MB once compressed.
Tutorials / documentation would really help a lot. There are a lot of users asking the same questions over and over again.
The library team will be focusing on transferring the library data to the new repository format before v5 - that is our priority and any “download tool” external to what is already available won’t be here for a while.
If anyone wants to step up to the plate and develop such a system - that’d be fantastic
Judging by the support questions asked here, I think the average KiCad users will struggle even with “basic” git. Git provides an awful lot of rope to hang yourself with. Certainly there are a class of users who will rather run a mile than attempt to use git.
A library manager that is a wrapper for git to do common functions is really essential I think. Advanced users can still run git commands directly.
The current symbol library handling does not allow easy git use, the new symbol library table will help with that. I would like to see other objects such as project templates, BOM scripts, footprints wizards handled in a similar (and consistent manner).
In tests, I found downloading a repo from github much slower than downloading a zip from KiCad server (https://kicad.org/download/source/ ). Github apply various bandwidth and rate limits - their service is designed for source code management, not for bulk deployment of data. There is one project that decided to exclusively deploy data via gihub, and ties up 5 of their servers with 100% usage. Github are quite generous there, but it still means users of that project experience slowness. We really want to avoid that!
Not really. The problem with the “just use git” is that it glossing over several important details, e.g. doesn’t update your KiCad library tables. Once you have installed git, and done some initial setup, then you can probably just do the git one liner, but that applies only to footprints. You still have to track renamed and new git repos.
Currently, Windows users can’t use git to update the symbol libraries because they are stored in c:\program files which requires admin access. Plus they need to separately install git.
Unfortunately I see people suggesting “simple solutions” which don’t address all the issues, and are much more complicated for the typical KiCad user than they make out. Don’t assume that all KiCad users are command line experts! Simple but unusable suggestions don’t help us to advance to a generally useful solution.
Bobc, you makes some excellent points regarding “simple solutions” and I am in agreement with you.
Having a Library Manager would simplify many of the housekeeping tasks required for libraries. This is essential for new users or even users with a skill level that is focused on electronic design rather than maintenance of git, as it could provide a more accessible manner of interacting with library data. Yes for many people who have worked out the kinks and quirks of Kicad from long experience and exposure, many of these things don’t seem like issues. But for many of the users(as can be seen in the forums) there are clear issues of usability for managing library data.
@bobc I’m not really trying to be flippant when I suggest using Git. Trying to assimilate all the different requirements that users ask for in these forums, becomes a very complicated problem. Any tool that someone develops will need to accommodate for local changes to files, handle file conflicts, serve compressed files and ideally only pull down file deltas. And this is only really the problem description. Implementing this would be months of work to get something of even basic functionality.
However, this is what Git already provides. Yes, I get that it is difficult to grasp at first especially for people who aren’t used to the concept of version control software. But it is still a valid option for a subset of users.
This is only true for the subset of KiCad Windows users that keep their libraries there. That’s not mandatory. It would be great to make the default install location somewhere else, and you or anyone else is welcome to either raise an issue with the dev team or implement a fix yourself.
Don’t think that I am against having a library management tool. I think it’s a great idea. But I am also keenly aware of how much work it would be. And I already spend way too much time thinking about KiCad.
Not only thinking. You are one fire lately. A lot of patches committed to the mailing list. (over a diverse set of new features.)
A lot of pull requests on the library.
The new KLC webside was done by you alone. (I only helped a bit.) You are also developing a shiny new lib download page.
Sometimes i wonder if you have any time left for anything but kicad.
This is one of the key improvements which will come with the new libraries - all footprint libraries will be stored in a single kicad-footprints repository. I have fought all year to make this change, so you are preaching to the choir.
And clear instructions to install components, footprints and 3D shapes. Nothing else.
I know people don’t like to read manuals or tutorials, we want everything running rightaway.
Unfortunately, magic requires a lot of effort from developers and many users are not willing to make some effort themselves.
If I use git or I download zip files it is not too important. Now I have both options.
The way kicad arranges libraries is a different question that will be improved in v6.
what do you mean by less efficient?
long story short you will need the full library downloaded if you want full offline acces. On the other hand you cant download the full library if you want to save the amount of data downloaded/stored. I find this a pretty good solution for people who dont want the full library (I would prefer downloading the full). The files should ofcouse be added to the local library when downloaded, this way building up the local library as the files are needed.
I’ll be the first one to agree that I don’t have a clue regarding the large scale view of things. As you correctly pointed out there are a lot of different users each with his own workflow and a lot of technical details. Thanks for pointing this out.
The zip download (available also on Github, and being developed for official Kicad page) will cover basic users. Yes a tool for setting fp-lib-table and symbols table would be really welcome. Is the fp-lib-table equivalent for eeschema coming with V5? If so, then fp-lib-table and symbols-table could be included in the repo and .zip file, which would result in quite clear instructions how to update the tables (copy them)
A user will download zip file multiple times, while with git only initial clone takes some time, from then it should be quite quick if you do it regularly
As for the installation of git, I personally don’t see it as a turn off. If you need advanced features in Kicad you need external tools (Freecad, …)
Also my view on git (not git in particular but on SCM tools) is that if you program even a little bit, this is a basic tool that you have to have at your disposal. Just like a blacksmith needs an anvil, only hammer and fire are just not enough. From my experience a lot of people designing circuits have some programming experience and would benefit from git. But before you point it out I agree that Kicad is not in the business of educating about SCM tools. I am just saying that if Kicad requires Git for advanced library management I don’t see this as a problem.
No it doesn’t, as I explained. I’m a software developer by trade, I know how to use git. I also deal with hardware designers who don’t know what the command line is. Literally, I had to show them how to launch a Command Window.
A Git interface is not a solution for the typical KiCad user
If we can’t agree on that, there is no point in further discussion.
Sadly, this will be another one of those threads that will run to hundreds of posts and not go anywhere, probably because developers unable to see it from a user POV, and users who really have no idea of the development side.
Git is not intended as the tool for the average user. (I don’t think oliver wanted to imply this)
The average user will use the download page and download a release of the lib. (as a zip archive) Not the current master via git. Yes it would be nice to have an installer that would unzip this, put the files into a central location and setup kicad to use these files. But i don’t think we will get that ready for the v5 release.
Git should only serve for people who want to be at the cutting edge or who want to contribute to the lib. (Yes for contribution management github and it’s pull request infrastructure is the best option in my opinion. Is it easy to understand for everyone? No of course not. But it is easy for us maintainers. And we are the bottle neck.)
