Hi,
Any one know what is the plan for put 1 symbol in 1 file(sweet symbol library)? It looks the V6 still uses one file to hold a symbol library.
this is a relevant link about this topic.
https://www.mail-archive.com/kicad-developers@lists.launchpad.net/msg33692.html
We can not just have .pretty but no .sweet 
TIA
It was kept as it is now for practical reasons, because of derived symbols. Otherwise it would have been more complicated. Itâs possible that it will be changed in some later major version, but I doubt.
We also have much more serious performance issues with footprints because large directories are very slow on WindowsâŚ
1 Like
Que
Interesting. Is this a file system thing?
TLDR; Linux keeps a top-level cache so that many things (like stat queries can proceed without actually touching each file. Windows does not. NTFS also has all sorts of layers for encryption/compression/etc that can add time. There are lots of other reasons but the upshot is that our approach in pretty files of having hundreds of small files is just about the worst thing you can ask Windows to do.
Hereâs some details from the horseâs mouth for those who are curious: https://github.com/Microsoft/WSL/issues/873#issuecomment-425272829
This has come up several times in this forum, the first display of a PCB takes a long time on a Windows PC with a traditional hard disk. The disk access times, the operating system and anti-virus software can make loading times in minutes. Windows tends to defragment files, but not directories.
SSDs help, but cannot eliminate the delay
How about using a zip file or some other cache system to contain all these small files to speed up the reading? Just like what has done to all the icon files? Anyway, keeping one symbol in one file is the way to go with git.
1 Like
Oooh⌠thatâs an interesting ideaâŚ
(But wouldnât git also treat the zip as one file? Or are there ways to make git zip-cognizant?)
One symbol in one file is not possible: derived symboles (AKA Alises) must be in the root symbol file, because they share most of root data, so âOne symbol in one fileâ makes no sense.
therefore this is not the way to go, even with git.
Footprints are a very different case: currently they have no derived footprints.
This is of course true only with the current system, and technically nothing is âmustâ. They could as well be in different files if one library would be one folder. For KiCad it would be just an implementation detail. I think the problem is what happens outside KiCad: itâs easier to keep the library under control when it canât be handled with the systemâs file manager.
That wouldnât matter if the libraries could be zipped locally. Also, thereâs already a stone age file format which isnât compressed but has several files in one bunch: tar. KiCad could support it transparently, I believe.
Remember to check https://gitlab.com/kicad/code/kicad/-/issues/6941.
Yea, enumerating a directory does not encounter any of the mentioned issues there. Itâs getting file attributes that could be problematic as you can cause round trip âfile opensâ multiple times. Depending on architecture, it can be pointless compared to just opening the file.
How about allowing more than one symbol/footprint in .kicad_mod files (maybe with some name cache in the header to speed up parsing)? This way one can squeeze an entire library into a single .kicad_mod file, stay git friendly, fix Windows FS slugishnessâŚ
Tom
1 Like
A strict âone symbol in one fileâ is problematic with derived symbols. (Derived symbols are the replacement for aliases, right?. I havenât played with 5.99 yet to see how derived symbols are handled differently than aliases.) But if the rule was âone symbol and itâs derivatives in one fileâ then it would be possible to have individual files, allowing someone to, for example, simply copy a downloaded symbol file from their downloads folder into the library folder and KiCad would find it. (This currently works for footprints.) As it is with library management in 5.1.x with aliases, if you copy an alias or itâs parent you get the parent and all the aliases copied as if the combination of the parent and all aliases are considered one âsymbolâ.
But I can see the value of one file per library to reduce filesystem churn, especially on M$ Winblows systems like my ownâŚ
Problem with tar is it doesnât support random access. Weâd have to scan the entire tar file at load and cache offsets to the individual components. zip is almost certainly better for this application. Itâd be cool if we could transparently support both folders and zip files for libraries.
Doesnât play great with git but not all libraries live in git. IMO, itâd be a fine way to distribute release libs.
How about using a zipping format without compression? (For example 7z and zip should support this.)
It still has binary data so not much point as you dont get vcs compatibility and dont get space savings either (however miniscule they would be).
What do you mean?
Apart from the binary parts in the beginning and end of the file itâs plain text. On the other hand distributed development doesnât work on this file because all changes affect the small binary parts and always clash. But this would work as a local solution to speed up the library loading, yet allowing a VCS for versioning/history.
I also wonder if there really isnât any text based format with a TOC, like tar where the metadata would be copied in the beginning of the file as lines of text.
Loading is a real problem, as was said SSD drive doesnât help much, my loading time is over a minute. And it happens quite often if the libraries are modified.
Exactly this. Most VCS donât even attempt to diff binary files and are not suited to store them efficiently.
The repo storage format and the released library format can/should be separate anyway.
2 Likes