3D Pin Headers Exceeds 1GIG: If KiCad Includes, Will Be Over 2 Gig

Layout 3D Models
42

I just tried a quick test on the my Connectors.3dshapes folder and 8MB compressed to 1MB with 7zip on Ultra, so compression helps but not as dramatic as some people think

43

As library sizes increase, the problem is going to get bigger (pun intended).

Focusing on STEP files and dropping WRL files would reduce it somewhat, but I don’t think that function should be the first consideration. Certainly compression and selective downloading would be the first techniques to try. Or reduce the number of updates on the library and release monthly. Or find someway of only downloading the parts that are new or have been modified.

44

Arraying items produces little to no computational overhead. I tested render speed of a 1x13 pin header (single model) compared to a 1x300 pin header (300 individual models) and the reported render speed was the same (a few milliseconds).

The greater the number of library items, the better the compression. This “backfires” with the use-case that people seem to want, which is individually accessible models. In such cases, compression doesn’t help anywhere near as much.

git pull origin master -f

This problem has been solved since the beginning. If you want to keep your libraries up to date and pull down file deltas, use Git.

1 Like
45

Perhaps this can be synchronized with each release.

A few questions:
Where is the exact repo(s) for libraries?
Does ‘master’ mean nightlies, the same as in KiCad?
Is there an equivalent git command for each stable release, or at least for 4.0.7 and forward?

Thank you in advance!

46

It can, and is (kind of).

Where is the exact repo(s) for libraries?

Currently the library repositories are a mess, let’s ignore the current ones and focus on how they will be for v5:

“master” means the main branch. The libraries do not maintain a “stable” and “testing” branch as that is too much work.

Is there an equivalent git command for each stable release, or at least for 4.0.7 and forward?

Yes. GitHub supports tagged releases. At each software release, the libraries get tagged.

Here is an example for the current kicad-library repository: https://github.com/kicad/kicad-library/releases

4 Likes
47

That looks simple and well thought out.

Any idea when it might be available to a nightly build?

48

The libraries are hopefully going to be disconnected from the actual software builds, and something that users can download separately.

As to when the data is available - we are in the process of transferring library data to the new repositories. As we do so, we are marking the old libraries to indicate that they will receive no further updates. However the current data will stay in place too.

3 Likes
49

My Data Plan will thank you!

50

That is true to an extent, more that some libraries are more “regular” and therefore contain more redundancy.
An extreme example is Pin_Headers.3dshapes, 560MB down to 23MB.

1 Like
51

With the developer power available I don’t think it makes sense to start developing library manager. I don’t think anyone can really estimate required effort.

Besides I think that Git and Github are quite an amazing solution as a library database. The only downside I see and it was already mentioned is file oriented setup which limits speed when you have a lot of footprints. This could be improved if footprints would be stored in one file per library as currently the symbols are. So for V6 I would seriously consider if switching symbol library layout one file per symbol is warranted. Otherwise Git and Github already cover most of the use cases. This will be even more obvious with sane repository layout in V5. Additional benefits, which come with using git and have not been widely advertised are:

  • pull request feature. This is a really great feature which I suspect that library maintainers are quite fond of. I don’t think the integration of new parts into the official library with proper inspection could be done any easier.
  • The ability to merge personal library with official library. If you clone the official library, make a local branch from master and switch to it, you will still be able to integrate any recent changes to master made by other contributors into your own branch. This is phenomenal property which really has not been widely emphasized (I suspect it will become more obvious with V5 repository layout). Doing it this way makes it much easier to use personalized libraries along the official ones. For example I don’t really like having footprint name visible in Fab layer. I clone the official library, send a script through library which makes footprint names non-visible and commit changes to my local branch. If any changes happen to existing footprints anywhere else I can (or better git can) merge this changes automatically.
  • Option to have multiple remote repositories. With git you can have multiple remote repositories. The official Kicad supported by Kicad library maintainers and an unofficial one, which could be open to anyone where people would be able to throw anything (as was already expressed in https://forum.kicad.info/t/ughh-the-current-libraries-are-a-mess/8026/29). Then it is up to individual user to choose from which repository he will merge to his local library. You can have additional repository which is available only within the department/company. And you have the same pull request feature to keep the department/company library under tight control (we are currently using this setup). We just need to show how this is done.

As for the size constraints I would package only symbols and footprints with the installer and keep 3D model available as a separate installer.
If the installers would have an option to select where to put the libraries and set the environment variables accordingly it would be even better.

What is really required is good documentation and a YouTube video or two to showcase how to:

  • use offline libraries without any git integration (I think this has been mostly covered), and it would be quite simple to show with help of installers.
  • simple git integration via one Git GUI tool (I’d recommend git-gui which is already a part of Git suite and is available for all platforms). This would show git clone and git pull to refresh the repository and maybe even how to switch to older version.
  • advanced git integration showing advantages of local branches, merges from master and different remote repositories.
3 Likes
52

@MitjaN what a great synopsis. As I am so invested in the libraries it is often hard to take a step back and develop a good summary, as you have done here.

You’re quite right, the current PR solution is pretty darn good. Especially now that we have checker scripts which do most of the repetitive work for us :slight_smile:

This needs to be advertised more, once you have it setup correctly it’s a very powerful feature.

Users are of course free to source libs from wherever they please, or create their own :slight_smile:

The footprint and symbol libraries once compressed are very small.

The entire footprint library set is 24MB once compressed.

Tutorials / documentation would really help a lot. There are a lot of users asking the same questions over and over again.

The library team will be focusing on transferring the library data to the new repository format before v5 - that is our priority and any “download tool” external to what is already available won’t be here for a while.

If anyone wants to step up to the plate and develop such a system - that’d be fantastic :slight_smile:

3 Likes
53

Judging by the support questions asked here, I think the average KiCad users will struggle even with “basic” git. Git provides an awful lot of rope to hang yourself with. Certainly there are a class of users who will rather run a mile than attempt to use git.

A library manager that is a wrapper for git to do common functions is really essential I think. Advanced users can still run git commands directly.

The current symbol library handling does not allow easy git use, the new symbol library table will help with that. I would like to see other objects such as project templates, BOM scripts, footprints wizards handled in a similar (and consistent manner).

In tests, I found downloading a repo from github much slower than downloading a zip from KiCad server (https://kicad.org/download/source/ ). Github apply various bandwidth and rate limits - their service is designed for source code management, not for bulk deployment of data. There is one project that decided to exclusively deploy data via gihub, and ties up 5 of their servers with 100% usage. Github are quite generous there, but it still means users of that project experience slowness. We really want to avoid that!

2 Likes
54

Not really. The problem with the “just use git” is that it glossing over several important details, e.g. doesn’t update your KiCad library tables. Once you have installed git, and done some initial setup, then you can probably just do the git one liner, but that applies only to footprints. You still have to track renamed and new git repos.

Currently, Windows users can’t use git to update the symbol libraries because they are stored in c:\program files which requires admin access. Plus they need to separately install git.

Unfortunately I see people suggesting “simple solutions” which don’t address all the issues, and are much more complicated for the typical KiCad user than they make out. Don’t assume that all KiCad users are command line experts! Simple but unusable suggestions don’t help us to advance to a generally useful solution.

3 Likes
55

Bobc, you makes some excellent points regarding “simple solutions” and I am in agreement with you.

Having a Library Manager would simplify many of the housekeeping tasks required for libraries. This is essential for new users or even users with a skill level that is focused on electronic design rather than maintenance of git, as it could provide a more accessible manner of interacting with library data. Yes for many people who have worked out the kinks and quirks of Kicad from long experience and exposure, many of these things don’t seem like issues. But for many of the users(as can be seen in the forums) there are clear issues of usability for managing library data.

56

@bobc I’m not really trying to be flippant when I suggest using Git. Trying to assimilate all the different requirements that users ask for in these forums, becomes a very complicated problem. Any tool that someone develops will need to accommodate for local changes to files, handle file conflicts, serve compressed files and ideally only pull down file deltas. And this is only really the problem description. Implementing this would be months of work to get something of even basic functionality.

However, this is what Git already provides. Yes, I get that it is difficult to grasp at first especially for people who aren’t used to the concept of version control software. But it is still a valid option for a subset of users.

This is only true for the subset of KiCad Windows users that keep their libraries there. That’s not mandatory. It would be great to make the default install location somewhere else, and you or anyone else is welcome to either raise an issue with the dev team or implement a fix yourself.

Don’t think that I am against having a library management tool. I think it’s a great idea. But I am also keenly aware of how much work it would be. And I already spend way too much time thinking about KiCad.

2 Likes
57

Not only thinking. You are one fire lately. A lot of patches committed to the mailing list. (over a diverse set of new features.)
A lot of pull requests on the library.
The new KLC webside was done by you alone. (I only helped a bit.) You are also developing a shiny new lib download page.
Sometimes i wonder if you have any time left for anything but kicad.

4 Likes
58

This is one of the key improvements which will come with the new libraries - all footprint libraries will be stored in a single kicad-footprints repository. I have fought all year to make this change, so you are preaching to the choir.

5 Likes
59

What do users need?

  • A server to download footprints from
  • A server to download components from
  • A server to download 3D shapes from

And clear instructions to install components, footprints and 3D shapes. Nothing else.

I know people don’t like to read manuals or tutorials, we want everything running rightaway.
Unfortunately, magic requires a lot of effort from developers and many users are not willing to make some effort themselves.

If I use git or I download zip files it is not too important. Now I have both options.

The way kicad arranges libraries is a different question that will be improved in v6.

60

What you describe is basically the download page currently in development by oliver. (Look at the mailing list for more details.)

61

what do you mean by less efficient?
long story short you will need the full library downloaded if you want full offline acces. On the other hand you cant download the full library if you want to save the amount of data downloaded/stored. I find this a pretty good solution for people who dont want the full library (I would prefer downloading the full). The files should ofcouse be added to the local library when downloaded, this way building up the local library as the files are needed.

2 Likes