Library management (3D and symbol)

Software
1

I have a couple of questions regarding library managemet.

When it comes to libraries of 3D models there are two libraries available. First one available at GitHub/packages3D which includes VRML as well as STEP models. The second one is a part of GitHub/kicad-library but has only VRML with some WINGS and/or FreeCAD models.

I plan to use GIT for our internal library management and with footprints it is really easy to include KIcad footprints from .pretty libraries as a submodule. The same goes for 3D models from GitHub/packages3D. But inclusion of symbols is difficult as they are not contained in a dedicated repository but are a part of GitHub/kicad-library](https://github.com/KiCad/kicad-library). This makes it imposibble to include it in your own repo as a submodule.

So my questions are:

  1. Does anybody know what is the status of each 3D library? Can I assume that the second one will not be updated anymore, and the first one is recoomended for new designs?

  2. Who should I talk to if I want to suggest puttiing (copying and maintaining the copy) the 3D models from GitHub/kicad-library in its own repo?

  3. Who should I talk to if I want to suggest copying symbols from GitHub/kicad-library in its dedicated repository and maintining the copy synced?

1 Like
2

The packages3d repo will be used in future. All models that are in this repo are currently also in the kicad-library/modules/packages3d directory.

I think @SchrodingersGat is searching for someone who can help out in programming everything needed to migrate the 3d shapes repo. (c++ programming is required.)

The migration will most likely come either in v5 or in v6 (If the code is not finished in time for the v5 release.)

Well this will be your own responsibility. But there is help availible. The symbol repo is taged for each kicad release. so you can just checkout the library tagged with your desired version number.

Also remember that a new symbol library format will be used in v6. This format will be similar to the current footprint format. (one file per symbol in a .sweet folder but not in separate repos per library because it is a mess to maintain that.)

1 Like
3

Can’t come soon enough really. This is what makes the KiCAD github resources look so messy currently.

4

Thanks for the info.[quote=“Rene_Poschl, post:2, topic:6118”]
Well this will be your own responsibility. But there is help availible. The symbol repo is taged for each kicad release. so you can just checkout the library tagged with your desired version number.
[/quote]

I have no problem setting the repositories the way I want. I can set up repos like this Symbols repo, Footprints repo. For footprints this works great. Official footprint repositories are included as submodules. This means I am synced with official repositories. Further more I can change official footprints (e.g. path to 3D models) locally and still merge changes on official repository to my local copy automatically and keeping the changes I’ve made (unless there is a conflict). It would be even better if there would be only one official repository, but that will happen sooner or later.

With symbols there is really no way to have the same setup. The official repo is fetched into (and not added as a submodule) and only symbols are retained. Consequently 3D files and other stuff is kept in repo history, making the repo big. Also repos are synced only locally and remote Symbols repo has no synchronization with official repo. I could sync them locally and upload the changes, but I would have to do this periodically. This calls for a dedicated machine and full automation. Not hard to do, but my thoughts were that this might be of use to somebody else also, thus it makes sense (at least in my head) that it should be located on official Kicad Github page.

Why all this hassle? I do understand that accessing libraries directly form github makes Kicad truly platform and installation independent. But I do prefer to keep my libraries local. Even if that means, that I have to deal with all the path issues. Now to sync the different machines I work on and to sync multiple developers Git is a pretty good substitution for a database. And since official libraries are available on Github I can connect (as as submodule or any other way) them to our library repositories. If I map local repository folder to a drive (on Windows) using subst and psubs I can also keep the paths the same on each machine (similar as in Kicad Classrom). Using Git hooks (post-merge) one could probably make this setup even platform independent.

TL;DR

Thanks for the info. I can see that given enough time Kicad will reorganize repositories in a saner way. Until then I will make do with my own solution. Everything else is just me talking about my process mainly to help me and maybe even somebody else.

1 Like
5

There may be GitHub size limit stopping all the 3d models getting merged into one single repo.
What would be really nice is a rsync like interface so that updating a library with one part modified doesn’t fetch the entire library. A massive saving of bandwidth and time

6

All 3d models are in one repo. In fact the symbols, templates are also in the same repo. (This is the problem that @MitjaN wants to address.)

What we currently do is split this up such that the 3d models have their own repo. (But again all 3d models in one repo.) To get the official repos setup this way would need someone who helps @SchrodingersGat with the programming of the needed parts.

There is only a limit on single file size.[quote=“davidsrsb, post:5, topic:6118”]
What would be really nice is a rsync like interface so that updating a library with one part modified doesn’t fetch the entire library.
[/quote]

Isn’t that how git normally works?

7

Yes, but it’s not how the gihub plugin works. The github plugin pulls a zip file of the whole repo, it does not actually use git.

1 Like
8

There is a python script in the kicad-library-utils repo that clones all footprint repos. Maybe this is the way forward?

9

there is also a nice tool to download only the desired sub-folder
https://github.com/MinhasKamal/DownGit/raw/master/res/images/downgit.png DownGit
https://cloud.githubusercontent.com/assets/5456665/17822364/940bded8-6678-11e6-9603-b84d75bccec1.gif
reference
http://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repo

1 Like
10

git was obviously designed for source code, where lines of code can be diffed and merged sensibly. Possibly git doesn’t make sense for data like 3D models, which are basically a BLOB, but I can’t suggest an alternative.

git also may not scale well for very large repos, and some suggest using multiple repos. There are some tools to help manage multiple repos, for example https://code.google.com/p/git-repo/

I did start looking into writing a git plugin for KiCad, but I’m not sure what the actually workflow should be when it comes to library management. I also wondered if KiCad should have a central vault of files, which the user can add/remove, as opposed to the current method of files spread around and the user supplies paths for KiCad to access them (or try to), which is quite messy and has proved to be error prone for some users.

1 Like
11

STEP internally is human readable, it describes the 3D model and everything - like code describes a program.
Should be no real problem for git to work it’s magic on it, no?
It’s not some binary blob, where without a translator/converter you have no clue what you’re looking at.

If by vault you mean, a root folder which contains all the other folders with user data (symbols, footprints, spice models, 3d models, templates, etc.pp.) then that’s a good idea - I already run my local systems like that.
If instead you mean some form of database, where there is need for a tool to get to the content (like a big zip archive) please no.

1 Like
12

That’s true, so maybe it’s a TLOB. I don’t think you can merge two different STEP files and get a coherent result though. The versioning only needs to be done at the file level. It’s a moot point I guess, there is probably nothing as easy and powerful as git/github.

Using the github API is not really a practical option for fine level access, the API is severely rate limited (10 accesses per minute) and only slightly better if you have login credential, which we would not want to mandate. The SVN access method has performance issues too.

I think a native git plugin is a better option, and also to avoid github specific APIs (other git servers are available :slight_smile: ).

Understandably there is push back on that idea, I’m not keen on cryptic archives either. The problem for KiCad at startup is that there is a mass of files, with no index, which may not even be valid data. So that means scanning the whole set to see what is valid data and index it. The advantage of an access tool is that it can update indexes and do validation, so KiCad can start up super-quick with a complete index of valid data.

The other question in my mind, is how to handle “user packages”? I would like to be able to “register” packages with KiCad, so it knows where they are without a lot of platform specific path mangling.

If you use Python, you probably use pip or similar to add remove/packages, so the process can be quite lightweight. The files can also be flat and text, so you could still go in and fix things with a simple text editor, or maintain via simple scripts.

Let’s agree on that! I’ve been trying to move towards the same idea. I think it can be implemented across platforms, but I’m not really sure. There are obvious packaging implications, some distros are strict about user data.

Anyway, it would be really nice to define KICAD_DATA_ROOT and then have everything in there.

1 Like
13

We have to plan for a future where there are many more schematic symbols, quite a few more footprints and far more (large) 3d models in both STEP and VRML. Keeping this lot up to date could mean GB per day of update without a fine grained modification mechanism.

Using standard APIs is desirable, KiCad has to be in a position to move away from a third party provider like GitHub if necessary.

14

I have been fighting for merging all the .pretty libs (there are currently ~90!!) into a single repository for a while, which has not been accepted as the GitHub plugin works does not allow subdirectories to be accessed. This means that one would have to download ALL the .pretty libs (as a ZIP file!) each time they want to access a single file.

There are a few other interesting* factors to consider too:

  • KiCad has a “Download 3D models” plugin that scrapes 3D models from the GitHub repo.
  • Instead of downloading a .zip file of ALL the 3D models, it downloads each model separately
  • The 3D model downloader uses the GitHub API (ish)
  • The GitHub API has been subsequently changed and the 3D model downloader can only download the first N models from each folder (all others are silently dropped)

So, KiCad currently has two different methods of downloading files from GitHub, both of which are… sub-optimal:

  • Downloading a .zip file for “live” access is really slow, and gives the user too much time to ponder the series of decisions that brought them to this point in life.
  • Utilizing the GitHub API to download files individually brings its own problems;
    • The API may change (it already has, IDK if the other devs know this, I only recently discovered it)
    • It requires a lot of web-scraping code
    • Rate limiting
    • May violate GitHub ToS (not sure)
    • Other OSS projects have had extra rate limiting imposed by GitHub for doing this very thing
  • Hard-coding a “GitHub” plugin without actually using git means that:
    • Non-GitHub git remotes are out of the question
    • Locked into GitHub
    • Don’t actually get any revision-management capability as users are just downloading static files, again and again and again and again.

Furthermore, it is a really bad idea to use libraries that can change at any time, at the whim of the sneaky sneaky librarians. Users should be able to control when they update their libraries, and be able to see what has changed!

The collaborative nature of the KiCad libs is really cool, and I think the quality is really improving, but I cannot fathom why users would actually use the GitHub plugin - I want to be able to exercise control over what changes and when.

I personally would like to see the GitHub plugin die.

I would be in favour of it being replaced with a git plugin, perhaps based around libgit or similar. This would provide many key benefits:

  • Users only download changes in files, huge speed and bandwidth saving
  • Not locked in to GitHub - e.g. could use company libraries that are managed in git
  • Not at the mercy of the GitHub API
  • Users can control when they update (each library could be marked with a * if there are updates available)
  • –diff messages could be displayed to show what has changed in the libraries
  • Probably some other benefits too (why restrict myself to only the benefits I can think of?)

I have begun to investigate implementing this, around many other commitments, so not much progress has yet been made. I also do not know how much traction it will receive, as some of the devs behind the current GitHub plugin seem quite attached to their work.

If anyone is interested in helping in this endeavour (in particular if you have experience with libgit?) then I’m all ears.

* for certain values of interesting

7 Likes
15

Good post, Oliver.

It is a pity someone put a lot of effort in the Github integration within kicad. It seems they wanted to help new users, but I think it doesn’t make life easier.

I don’t use $(KIGITHUB). Never.

I need a place to download the libraries from. Nothing else.

Once in my hard drive, I can manage symbols, footprints and 3D modules easily. As many times said, Andy being an expert saying this, people should use their own libraries. And starting from the official libraries is the best way to build up a good own library tree.

In my particular case, I create a specific library for each project. Very easy with the new .pretty and 3D folders, harder with the schematic symbols. Looking forward .sweet system…

1 Like
16

One issue with all software is that there are always at least ten ways to do one particular thing. As we can see with libraries. Most of us have come to a point to appreciate having libraries under strict control. Someone has libraries for each project separately, one has them in one folder and I also have them in one folder where each subfolder (symbols, 3D and footprints) is under git for revision control (and sync with official libraries).

As for obtaining the libraries, somebody uses github plugin, one uses integrated git download tool to keep libraries local, one would prefer to use rsync, and I use git (clone, fetch, merge, submodule).

I think that it is pretty obvious that there is no way KiCad could support all the different processes. Moreover as KiCad is a collection of different tools each handling libraries differently (I know this is announced to change and I think that we all wait for this day to come).

The idea to switch to libgit2 backend is a really, really smart one, but I can only support you with words, as PC application development is not really my forte.

As for the library management, it would be nice if KiCad would push users a bit more strongly toward one way how do things. But I do not have a clue how to do this neither which is the proper (best) way.

All this discussion could also be ended if Kicad moves libraries to proper database. Then there would be only one and proper way how to do things. But I think nobody wants this (users and developers)

17

A method that notifies you that a symbol/footprint/model you already are using has been updated and allows you to sensibly investigate why would be ideal.

The saving in bandwidth is important to the users to to the host. I can see KiCad getting into trouble over TOS as the bandwidth consumed grows. I am sure that the number of active KiCad users has increased significantly in the last 12 months.