We have scripts ready to collect and merge all the libraries, this part is not the difficulty 
Let’s not turn this into a bike-shed issue on how the libraries are arranged on GitHub. This has already been a topic of discussion among the librarians (and other interested parties) with hundreds of comments and hours of back and forth. Let’s keep this pertaining to “how do users access and update KiCad libraries”.
2 Likes
For the most part, I create my own symbols and footprints.
Where I get hung-up, is the 3D models. At first I thought the 3D models were just a gimmicky thing; now I like them as a “sanity check”.
I do not know how to download individual 3D models from GitHub, so I use the wizard in PcbNew, and waste my data re-downloading every 3D model every time I think there may be an update that might be beneficial to me.
Every download of KiCad Windoze build, stable or nightly, also downloads mostly everything I have already downloaded several times this year; most of it I doubt most users will actually use in new designs.
I am mobile, so I use my phone as a HotSpot for my computer to connect to. The current state of KiCad means that I have to choose how my Data is used.
[quote=“SchrodingersGat, post:51, topic:7625”]
The git option provides you with the diffs you need
[/quote]Hmm… That Torvalds guy gets smarter all the time. So basically the installer should just run the appropriate ‘git’ command? Sure seems easy enough.
As I recall, the issue hinges around having a better update system than the github plugin. Git isn’t very good as a deployment system, but there are not really any off the shelf systems I can think of. Just about every software package seems to develop their own bespoke updater. Python has pip, .NET has NuGets, all the Linux packagers etc.
Arduino had a similar problem with multiple cores and libraries bloating the download of the main IDE, and also a rather confusing mess of libraries and versions. They came up with a fairly simple and lightweight updater which actually works quite well, and is easy to manage for Arduino, users and third party providers. I have created a couple of core packages for Arduino, and it is really quite easy.
I think it would definitely be worth looking at and seeing how it could be adapted to KiCad.
1 Like
It seems it should work well for Windows as long as you don’t move the libraries from the default location. Check to see if the library structure exists. If it does, update, if not create (with permission of course). Even Linux is trying to be a little more uniform in the file system between distros.
Yes this is precisely the issue that I want to address.
There will (when I get around to finishing it) a library website where you can browse the models / footprints and download the ones that you want. You should no longer have to use the weird 3D model downloader from within KiCad
That’s what I would want also. The ‘github plugin’ is a very poor implementation (especially as it doesn’t even use git functionality. The complication is making it work across multiple OS with different default paths etc. Not impossible by any means - and arduino is cross platform anyway.
1 Like
If you agree on my listing of steps, I can experiment with getting git read access with python. Then we’ll be able to take the next step. I’ll let you know when the aggregated libraries would be useful. I’ve got a big project due end of month, not sure how much time I’ll be able to spend before October.
Edit: I’ll first take a look at the wizard. It might be in python already. The python got library has a few h hierarchical dependencies that are all pure python (AFAICT) so that should ease importing those libraries.
Does all that sound ok?
Greg S.
I did start writing a library manager/downloader app, using libgit2 and also some web scraping. It’s a prototype, it’s not cross platform, and it’s not C++ so pretty useless for inclusion in KiCad, but I wanted to get my head around what sort of GUI and operation would look like. It also only works if KiCad is installed in a user location, ie. not Program Files.
For example, one issue is it’s hard to tell what the current installed versions are, because the Kicad files lack metadata. I was going to solve that by writing some metadata files, which would be valid after the first download using the app.
git is quite inefficient for the 3D libraries, it’s 2.5G of data which even on my super-fast fiber takes a while to download, and even copying on disk is quite slow . A “thin” clone would be better, but in that case one may as well pull the zipped master and just make a note of the head, in order to check for updates.
I don’t think there is a really simple way to handle large data blobs like 3D models that is efficient, you have to get into caching, on demand decompression etc.
3 Likes
The KiCad master branch (nightly) has a script for using git to download the 3D models. That’s a bat file, but there is probably equivalent for Linux and MacOS.
Edit: probably requires git to be installed and in the path.
Is this as simple as changing the git clone to git pull?
If the user has made any local changes to the library, then any git commands can become problematic. It is not a simple problem to solve for users who do not want to learn to use git
In the future we should have 1 file per object (symbol, footprint, 3d, etc.) and when the user changes it, the date of last modification must differ from the last global update (via whatever means) that should be fixed in the creation date of the file.
Just don’t overwrite files like that, unless the user authorizes it, would be my first ‘reaction’…
So the global update sets the last modified date and creation date at the same point for every file it puts on the disk of the user.
Next round it compares creation/last mod and if they differ the file won’t be touched.
Is that safe enough?
Any OS out there that doesn’t hold this information for a user land script?
I know the consistent recommendation is that users should not change provided libraries. Is there any way to enforce this?
If modification is prohibited, does that “solve” the git problem? If so, then that seems to set the stage for implementing git commands in python. It might also be possible for python to determine if the file is modified. And if so, warn the user before overwriting the library. The python code could check prior to git pull to avoid git complications?
Modification times could be problematic, but hashes might be used.
- download new file
- locally store initial hash (SHA-2 or similar)
- on update, verify local hash and
- present dialog for all unmatched.
- remove (or store backup) authorized files.
- git pull all that match hash (I.e. locally unchanged files) and all authorized files.
- restore backups for failed git pull
How granular do you want this process to be?
Type of lib level - all symbols, all footprints, etc.
Sub-library level - all Resistor symbols, all SOT packages, etc.
Single object level - ATMEGA328P_32pin symbol, 0805 SMD hand solder footprint, etc.
?!?
The higher up you stay the bigger the downloads and the less sophisticated the interface.
The further down you go, the more sophisticated this has to be and most people probably don’t want to bother with a high level of granularity anyway…
Maybe I ask the wrong question though?!
1 Like
You’re working on implementing a high granularity automated process it seems.
Right now, it would be as granular as the git commands in the bat file I posted upthread a few posts.