@stambaughw
Could you explain how library cache and rescue symbols functionality work now? It looks like there are some considerable changes here.
If I recall correctly the rescue window used to appear when there were differences between cache and external library. Now it appears only when required library is not found, otherwise symbols from the external library is used without any warnings and original cached symbols are overwritten with next save.
If no external libraries were lined to project then cache used to be used. Now in this case rescue window appears, rescue library is created from cache and added to local library table. Is looks like after using these rescued symbols library and saving the file there is no way back to using external library. I mean that you cannot easily synchronize schematic with external library because you’ve lost library names (which are now required).
Am I correct? Is it expected to work this way? It seems to be more dangerous for projects (as it was before introducing rescue functionality).
The symbol rescuer had a potentially fatal flaw. It only rescued symbols that were different than the symbol in the original library. It did not rescue symbols could no longer be found in any library. This meant that you were always using the cached symbols in this case. This was a serious flaw. If the cache library ever got corrupted or deleted, all of the links to the symbols that no longer existed in any library would be broken. This is not the purpose of the cache library. It only should have ever been used as a temporary backup in case the user accidentally deleted a symbol in the schematic while editing. Unfortunately, it has been used a crutch since the beginning of the project albeit a crutch that was hidden from the user for the most part.
To answer your question, it makes perfect sense to rescue symbols that cannot found in the library list or now in the library table to prevent the situation describe above. This way if the cache ever gets corrupted or deleted, there should not be any broke symbol links (unless of course you delete symbol libraries that have links in the schematic) because they are always rescued to the rescue library. So your assumption about it being more dangerous is incorrect. It is now far less likely to have a broken symbol links.
The risk that I’m afraid of is mainly that symbols that are loaded to schematic may differ from those that were used while drawing and now I won’t get warning that they may not match.
I use my own libraries and sometimes I change them (names, paths, pin positions…) so when I open old project I can’t rely on links to my libraries and I usually rely on cache. Now probably I should rely on *-rescue.lib created from cache after removing sym-lib-table entries.
You are far better off with the changes I made than you were without them. We should have never relied on the cache and should have displayed a warning to users that there where symbols in the cache that no longer existed in any library. If you changed a symbol in a library and it doesn’t match the one in the cache (which should be a copy of the last symbol pulled from said library when you save the schematic), then you want to use the rescued symbol. If the library symbol was used, the schematic would be broken by the changes to the symbol. This is how the rescuer always worked. All I did was include rescuing symbols found in the cache that could no longer be found in any library. The rescue library is far safer than the cache.
While waiting for the windows nightly to be updated I am looking for some clarification: In the blog post on the KiCad web site under point 3:
Make sure your default KiCad template project file (kicad.pro) has no entries in the [eeschema/libraries] section or get a copy from the KiCad source repo. If you build from source or are using a nightly build, the update project template file has already been installed. It is imperative that you do not have any symbol libraries in your project files. Otherwise, you could possible be effect by this bug which the symbol library table is designed to resolve.
(Emphasis by me).
What does “have any symbol libraries in your project files” actually mean? Does it refer to the settings managed under Settings- Component libraries that is stored in the project .pro file? So should the list of component library files be cleared out before remapping?
Also under point 5:
For the most accurate remapping you should create a project library by copying the project cache file (project-name-cache.lib) to a different file and add it to the top of the symbol library list. You must use a version of KiCad prior to the symbol library table implementation in order to do this.
If one has made sure that there are not any symbols that only resides in the cache as instructed under point 1 of the blog post, will step 5 add any additional information, or is it just a safeguard in case something has been forgotten? (It just seems that this could contribute to a messy sym-lib-table file.)
One manual way to migrate (using the own internal language of the Eeschema files):
Backup your project first, just for safety.
Copy your “-cache.lib" as "-rescue.lib” (as example);
Open the *.sch in a text editor;
Replace strings in all the text "\nL " to “\nL Project:” (\n in the indication of new line, so “L” is in the begin of the line);
Open the schematic in the new Eeschema but cancel all the automatic migration;
In the symbol table import your “*-rescue.lib” as “Project” in the local (project libraries);
Restart Eeschema / KiCad;
Re-open the schematic (probably you be not asked you to synchronize the library).
I used this method tho migrate a 9 files (hierarchical) project and had almost no issue (the only founded is related to one new version of a MCP4922 in the internal library that use “/” in the name and create some problem in the cache interpretation). With my old projects with just 1 schematic file I had no error at all.
I had to do manually, because the automatic procedure couldn’t deal with this big project. And this procedure as inspired in the step 5 of https://kicad.org/post/symbol-lib-table/.
Is slash ("/") allowed in part names now? The most recent KLC prohibits slash, but there are many parts in the official libs that do use slash (notably Microchip parts). (See https://github.com/KiCad/kicad-library/issues/1799)
The slash has been allowed in the past. We disallowed it only in the latest reincarnation of the KLC in preparation for the new file format. (In the hope that no symbols with this char in their name will be left at the time the new file format comes along.)
Exactly, it was a problem with a MCP (Microchip) and in the older library, before 10/Nov/2017, they used “_” instead “/”.
About the script, yes, could be made. Even step 4 tp 5 is just to create the file “sym-lib-table” in the content:
(sym_lib_table
(lib (name Project)(type Legacy)(uri ${KIPRJMOD}/myName-rescue.lib)(options “”)(descr “”))
)
Yes. You can test this by simply creating a new project from KiCad.
Open the project (.pro) file of the newly created project with a text
editor and go the end of the file. You should see a section entry:
[eeschema/libraries]
There should be no “LibName1=some_lib_name” entries after this section. This will only be an issue if you use a custom default “kicad.pro” file from a non-standard installation path using the KICAD_PTEMPLATES environment variable.
If you followed step 1, then this step wont be necessary. I fixed an issue with the rescuer which should mitigate this issue but this step is the safest way to do it. I was trying to make sure that there are minimal issues with the remapping. I would rather the remapping be accurate as possible. The remapping will only add one additional entry into the project symbol library table in this case since nothing should require rescuing.
Thanks for the clarification and confirmation Wayne.
As I make heavy use of custom libraries, and for my own libraries have used a mostly one symbol per library approach, all residing in a folder “own_libs” under my home directory (I also use a custom location for the official Kicad Libraries all stored locally etc.), I am interested in what to expect forward with respect to how things will be organised when also the new symbol format will be introduced in ver. 6 (if that has been decided yet). Can we expect the equivalent of the “pretty” format for the footprints where each library is a folder, and each symbol is a different file?
(My number of custom libraries and projects are still at a level manageable for manual reorganization, so this might be a good time to think about how things will end up. If a one symbol per file approach is expected in ver. 6 then my current strategy might be redundant and create unnecessary overhead.)
[In the official KiCad library there is a “symbols” folder with files having a .sym extension. I wonder if this is related to a new file format, or what they are?]
This has nothing to do with the new lib format.
As far as i can tell these are just exported graphics. (Symbol editor right toolbar has an export button. This button exports sym files.)
I’m trying to follow and understand this thread and the whole library system, to no avail. I have a couple of small projects and finally I had to fix them manually by opening the symbols from eeschema using right click on symbol -> Properties -> Edit Properties -> Component Name. I found out that when the automatic rescuing succeeded the name is in the form library_name:component_name instead of just component_name. And both the lib name and component name are “rescue”-something. That works but doesn’t look nice so I changed them manually even for the components which worked after the automatic process (and mostly they didn’t work). First I of course had to fix the project-specific libraries in Manage Symbol Library (Schematic Library Tables).
Then I started to wonder why this two-stage (or three-stage, or four-stage, depending on how you look at it) process at all in such a way. Basically what I want to have is libraries I have used previously in my project to be used in a new way, and the components I had to be prepended with the new library names. So that intead of having
/path/to/my/library/folder
in the symbol library manager I would have
/path/to/my/library/folder/libname.lib
And in the schematics instead of
symbol_name
I would have
library_name:symbol_name
And I wonder why that can’t be automatic and why there are two dialogs which then convert my symbol libraries and names to the form which has “rescue” in them instead of just having one dialog and automatically changing old symbol names to new library:symbol names.
But I never understood the library system to begin with and didn’t get it to work like I wanted. In the new system it seems to be easier with new projects. Previously I ended up adding new libraries in the old Component Libraries dialog directly by adding a User defined search path, instead of first adding library folders in the Manage Symbol Library (Schematic Library Tables) dialog. Maybe that messed up the migration system so that rescuing didn’t work well.
And if you didn’t notice it already, I didn’t understand almost a word about the directions given in the blog post.
Anyways, KiCad is going into a good direction, it’s exciting to learn to use it and to follow where it’s going and I wish I can help development in the future.
Yes, the new symbol library file format will be similar in design to the current footprint file format. Symbols will be one symbol per file for the most part. There may be some exceptions.
Rescuing is not remapping. Rescuing occurs when the symbol in the cache is different from the same symbol found in the library or the symbol in the cache cannot be found in any library. Remapping maps the library symbol links of the schematic symbols to the new library table.
Prior to my commits on Friday, the symbol library table dialog had no effect. It was just part of an incremental development path.
