Schematic Symbols Everyone Is Forced to Download

Schematic Library Symbols
42

Why aren’t the wrl and step files individually compressed during the creation of the installer and then left compressed while installed?

Compressing each file with xz (same algorithm as 7z), I got a 90% reduction in total file size: from 2,7G to 280M.

Obviously the model-loading code must be modified to decompress the models on the fly, but given the huge benefit in terms of bandwidth and space reduction it definitely makes sense.

If you want to test it yourself, use these commands:

$ cd kicak-library/modules/packages3d
$ find -type f -print0 | xargs -0 -P4 -n 1 xz
1 Like
43

Solid compression, as described by @keruseykaryu above, can spot reuse of blocks between various step modules and remove this redundancy. Pre-compressing would prevent this, which is also why compressing the exe has little effect

44

Solid compression is not much better than compressing the individual files:

  • unpacked files: 2.7GB
  • packages3d packed and compressed (tar.xz): 221MB
  • single files compressed individually (xz): 280MB

The drawback of solid compression is that it only helps with download sizes and cannot be used to reduce the disk space required to store the 3D models on the user’s hard drive. (Why? Because seeking into a compressed 221MB file is much more complex and slow than just decompressing a single 20KB compressed model file.)

Personally, I’d suggest to compress all the files individually (at least on the user hard drive) and write a tiny shim that decompresses the 3dmodels on the fly before using them.

45

True, but disk space is rarely an issue except on old PCs and for SSD users. Modern PCs come with 1TB as a minimum.
Data caps are a real issue for many users and server bandwidth is an expense for the project.

2 Likes
46

The devs/maintainers must have this on the radar, no?
That’s why I would get this on the dev mailing list instead of the bugtracker really.

1 Like
47

I agree that the installer’s compression options are incorrect.
The installer used is nsis (I also use it) and the compression method used in version 4.0.6 is deflate, which is the default method, is fast but almost does not compress.
It is easily solved by placing the following line at the beginning of the script
SetCompressor /SOLID lzma
I did a little script to test the compression:

SetCompressor /SOLID lzma
Name kicad
OutFile kicad_test.exe
InstallDir "$PROGRAMFILES\kicad"
Section kicad SEC0000
    SetOutPath $INSTDIR
    File /r "C:\Program Files\KiCad\*.*"
SectionEnd

Conclusions:
kicad-4.0.6-x86_64.exe -->426MB
kicad_test.exe -->270MB

Nsis compiler output
Using lzma (compress whole) compression.

EXE header size:               32256 / 34816 bytes
Install code:                          (1013811 bytes)
Install data:                          (1282820347 bytes)
Compressed data:           283837184 / 1283834158 bytes
CRC (0x6D7C86D6):                  4 / 4 bytes

Total size:                283869444 / 1283868978 bytes (22.1%)

A reduction of 156MB. In the next version the compression must be greater since there are more number of text files.

2 Likes
48

Hi everyone, great discussion here :slight_smile:

There are some really great points here. The current methodology for retrieving libraries is insane. The KiCad Way of managing libraries differs for each type of library (symbol, footprint, 3D model) and each way is (IMO) terrible.

I have spoken at length about this on the dev forums and also on here, so I won’t rehash that (too much). My vision for how things should be managed:

  1. Separate software installed (KiCad program) from library installed
  2. Provide (ideally as part of the installer) option to download libs (symbols / footprints / 3D) (compressed!). .These will be the “latest stable” version of the libs
  3. Users who wish to track the libs on GitHub can do that (using existing Git tools, not KiCad)
  4. Remove the “GitHub plugin” from KiCad, and thus increase quality of life for many KiCad users
  5. Merge all the footprint (.pretty) repositories into a single repository
  6. Provide an official “KiCad libraries” website. Here, users can either download entire library-set (zipped) or individual libraries / 3D models. These will be based on the GitHub libs and updated every week or so.
  7. (Optionally) provide a KiCad updater which can update KiCad version without re-downloading the entire package.

This will reduce the bandwidth burden greatly.

But why has this not happened yet?

a. The separation of the libs / installer needs sign-off from the devs / package maintainers.
b. Some of the devs are really attached to the GitHub plugin.
c. Time / effort / life

I am currently working on generating a “libraries” website. This will provide downloads for the libraries, and also a better “how to download” section, as well as a “how to contribute / KLC” section. My hope is that when this is operational, I can convince the devs that we should be making life easier for the users, and not forcing them to download gigglebytes of data (that rarely changes)

So, what to do for now? Start a bug report that the installer should not always package the libraries (but should provide a way to download them). Everyone who feels strongly about this should add their voice to the bug report by pressing “this bug affects me”.

3 Likes
49

If you really want to be able to decrease bandwidth you should probably look at the ability to do ‘diffs’ on a time stamp basis.

1 Like
50

Can anyone clarify what this means?

51

If users are provided two options:

  1. Download highly compressed set of libraries (one-time, or when they want to update)
  2. Clone the git repo

then this should be sufficient. The git option provides you with the diffs you need :slight_smile:

1 Like
52

I can!

The official KiCad libraries are on GitHub.

The (many) symbol libraries are provided as a single repository.

However, each footprint library (.pretty) is a separate repo. Currently there are 105 footprint repositories. Each time a new library is added, a new repository is added. A much better way would be to simply add a new directory to the footprint library repository.

Currently, if a user wants to track the github repos, they need to clone / fork a new repo every time a library is added. It also means that the librarians have to manage commits across over 100 repositories.

We would like to merge all the footprint library folders into a single repository. Then, users can track e.g. github.com/kicad/footprints and have access to all footprints.

Does this make sense?

2 Likes
53

The latest typhoon near Taiwan has cut several Internet cables, making the Net crawl in SE Asia. The 700MB Nightly is painful to download.

1 Like
54

I would highly recommend raising a bug about this and getting as many people on board as you can. The lead developers do not generally read this forum, and it is they that hold the keys to the kingdom.

55

I think I got it; but I am NOT a GitHub user for the most part; only with KiCad. However, I am an above average computer user, and I’ve been hobbyist programming since very young. So, when this stuff gets difficult for me, I know the majority is really suffering.

I also suggest that the (.3dShapes) folders are included in the same repository. And, with the same name if possible. This may already be in the works and I just can’t see it from where I am at as a user.

Thanks for the info.

56

I don’t see how the 3D shapes can be the same name. There are multiple 3D shapes for each footprint. An easy example are through-hole pin headers. They can be straight or 90 degrees. There can also be a plethora of modules that go into pin headers (nRF radio transceiver reference modules, for example). So this is a many to one relationship, perhaps even a many to many.

Disclaimer: I am not an expert nor a library maintainer.

57

Can this be done experimentally outside of the official repositories and source code?

  1. collect all repos in one place.
  2. write sync functions in python
  3. provide dialog with python
  4. package into Action Plugin

I’ll look to see if #1 is possible, I know the others are.

58

Removing the .WRL files would break existing designs. They need to be retained for backwards compatibility.

59

Simple google search shows the following steps to merge git repositories. Where is the list of 3D repos?

The basic idea is that we follow these steps:

Create a new empty repository New.
Make an initial commit because we need one before we do a merge.
Add a remote to old repository OldA.
Merge OldA/master to New/master.
Make a subdirectory OldA.
Move all files into subdirectory OldA.
Commit all of the file moves.
Repeat 3-6 for OldB.

60

Indeed your examples are a subset. But, why would you not put this subset inside the primary folder?

Where else would you suggest they migrate off into whatever other location in space?

61

Library management is very opaque at the moment which is a big problem. There’s not really any info on the KiCad website on how to download / manage libraries. IMO this should change, and this is my focus currently (outside of work, and you know, life).

1 Like