A strict “one symbol in one file” is problematic with derived symbols. (Derived symbols are the replacement for aliases, right?. I haven’t played with 5.99 yet to see how derived symbols are handled differently than aliases.) But if the rule was “one symbol and it’s derivatives in one file” then it would be possible to have individual files, allowing someone to, for example, simply copy a downloaded symbol file from their downloads folder into the library folder and KiCad would find it. (This currently works for footprints.) As it is with library management in 5.1.x with aliases, if you copy an alias or it’s parent you get the parent and all the aliases copied as if the combination of the parent and all aliases are considered one “symbol”.
But I can see the value of one file per library to reduce filesystem churn, especially on M$ Winblows systems like my own…
Problem with tar is it doesn’t support random access. We’d have to scan the entire tar file at load and cache offsets to the individual components. zip is almost certainly better for this application. It’d be cool if we could transparently support both folders and zip files for libraries.
Doesn’t play great with git but not all libraries live in git. IMO, it’d be a fine way to distribute release libs.
Apart from the binary parts in the beginning and end of the file it’s plain text. On the other hand distributed development doesn’t work on this file because all changes affect the small binary parts and always clash. But this would work as a local solution to speed up the library loading, yet allowing a VCS for versioning/history.
I also wonder if there really isn’t any text based format with a TOC, like tar where the metadata would be copied in the beginning of the file as lines of text.
Loading is a real problem, as was said SSD drive doesn’t help much, my loading time is over a minute. And it happens quite often if the libraries are modified.
The idea would be to use an unpacked format for library authoring (i.e. the official library Git repository would be unpacked) and a packed format for distribution (to make KiCad faster). The packed format would likely also be compressed at that point.
As @retiredfeline says, there is really no point to forcing the libraries to be in the same format for editing as distribution. Related: it’s also desired to allow embedding 3D models inside footprint libraries, etc. Distribution libraries should be optimized for performance and file size, meaning they should be single-file and compressed. VCS diffing of these distribution libraries is not a relevant feature in comparison.
That’s right. The only problem for me is that reloading the footprint libraries takes so long in KiCad when something has changed. I suppose that when the library system is redesigned I can have compressed official library and non-compressed personal library which I can edit easily and keep in a VCS.
I guess it depends on the direction we are going to go with library loading in future since we do need a take a serious look at the increasingly significant loading delays for users that aren’t rocking a 5900X, multi SSD beast like I do.
I have nothing in this exercise, but I wonder if the effort to make the changes discussed in this thread are the best use of available resources? I am running KiCad in a Windows 10 Pro virtual machine on a Dell Vostro 3681 (I5-10400 processor) PC running Windows 10 Pro equipped with 32GB ram and a 500GB M.2 NVME and it is so much faster than the previous real PC it ran on it is incredible.
Just thinking that the inevitable hardware advances may negate the need to completely redo the symbol library format. Just my $0.02.
It’s not just about the user’s computer speed, it’s also about download size and the ease of maintaining the official library (and a desire to have harmonized concepts in the symbol and footprint libraries)
Slowing OS software performance seems to be quite capable of matching hardware speed increases.
Add to this that the industry is trying to push everything into the cloud, which means that each file access might cost seconds across the oceans. We have already seen the developers trying to speed up KiCad performance on a LAN
Nah this isn’t a cloud issue. NAS/shared drives in corporate networks are defacto standard for decades. That’s my focus with the fixes. Granted, at my work I’ve long since implemented a 40 gigabit backbone to our Windows Server driven storage and the user switches, so latencies are no issue (anymore…it used to be a 1 gigabit pipe between our entire VMware/SAN cluster and the user switches, I cried everyday). But not everyone in corporate environments is fortunate and they do sometimes get the short end of the stick, even being physically in a building only feet from the server because their IT is…not very on top of things. The fact it fixes cloud drive storage issues are a bonus since it is all related.
