Symbol filtering sort order

[topherbuckley]

Is anyone familiar with how KiCAD sorts the filtered list when finding symbols in eeschema? It doesn’t appear to be alphabetical, so I’m a bit confused as to how it is sorting things.

This question arose after I had used the pspice diodes throughout my design only to find that their polarity is opposite from all other diode symbols, and opposite to the actual footprint of my part. This led to a manufacturing error only discovered after pcb assembly…ouch! See more here

In hopes of saving others from a similar oversight, I wanted to know if the search filter sorting could move the SPICE symbols to the bottom or at least not the top (first default pick) for everyone. Maybe this is even intentional to solve oversight for those using NgSpice?

[JeffYoung]

We use a scoring algorithm:

void LIB_TREE_NODE_LIB_ID::UpdateScore( EDA_COMBINED_MATCHER& aMatcher )
{
    if( m_Score <= 0 )
        return; // Leaf nodes without scores are out of the game.

    if( !m_Normalized )
    {
        m_MatchName = m_MatchName.Lower();
        m_SearchText = m_SearchText.Lower();
        m_Normalized = true;
    }

    // Keywords and description we only count if the match string is at
    // least two characters long. That avoids spurious, low quality
    // matches. Most abbreviations are at three characters long.
    int found_pos = EDA_PATTERN_NOT_FOUND;
    int matchers_fired = 0;

    if( aMatcher.GetPattern() == m_MatchName )
    {
        m_Score += 1000;  // exact match. High score :)
    }
    else if( aMatcher.Find( m_MatchName, matchers_fired, found_pos ) )
    {
        // Substring match. The earlier in the string the better.
        m_Score += matchPosScore( found_pos, 20 ) + 20;
    }
    else if( aMatcher.Find( m_Parent->m_MatchName, matchers_fired, found_pos ) )
    {
        m_Score += 19;   // parent name matches.         score += 19
    }
    else if( aMatcher.Find( m_SearchText, matchers_fired, found_pos ) )
    {
        // If we have a very short search term (like one or two letters),
        // we don't want to accumulate scores if they just happen to be in
        // keywords or description as almost any one or two-letter
        // combination shows up in there.
        if( aMatcher.GetPattern().length() >= 2 )
        {
            // For longer terms, we add scores 1..18 for positional match
            // (higher in the front, where the keywords are).
            m_Score += matchPosScore( found_pos, 17 ) + 1;
        }
    }
    else
    {
        // No match. That's it for this item.
        m_Score = 0;
    }

    // More matchers = better match
    m_Score += 2 * matchers_fired;
}

[JeffYoung]

If you don’t read C, basically an exact match of the name is highest, followed by early matches within the name, followed by early matches within the keywords.

[Rene_Poschl]

The pspice lib really is only there for backwards compatibility. It should really be removed for v6. The new simulation_SPICE library takes over its role.

There is however nothing we can do if users do not notice the word Simulation in the library name plus the MASSIVE disclaimer in the symbol description.

[topherbuckley]

I see. What about just renaming the pspice.DIODE to pspice.D and Simulation_SPICE.DIODE to Simulation_SPICE_D similar to how it is done in plenty of the other libraries? That would bump them further down the list without touching the otherwise clever sorting going on there.

[topherbuckley]

I completely agree that this was my fault, but I wouldn’t generally consider myself to be someone who overlooks details, so I’m sure many others will suffer my same fate. Just trying to help them out if possible. I believe its more to do with people being unfamiliar with the norms of KiCAD or schematic design in general (myself included). There are a LOT of details in the whole KiCAD environment one COULD pay attention to, but there are plenty that people, especially when just getting started with it, will ignore or overlook as there is simply too much to take in all at once. I went through word-for-word all the online docs and tutorials, and still made this mistake, so I’m guessing there will be plenty more people who give it even less diligence.

To be fair, the description on these is well documented, its just very easy to overlook if your window does not display this info by default (screenshot of what I see by default attached).

Again, yes, this is my fault, but if there is a way to better the UX of KiCAD, I don’t see a reason to ignore it simply on the basis that those already experienced would never make this mistake.

[chschlue]

This is more of a workaround than a solution but if you’re not going to use the simulation symbols at all, you could just remove those two libraries from your sym-lib-table.

[SembazuruCDE]

Or if you really don’t want to completely remove them, simply un-check the active column for them in the symbol library manager. Then if you need them again re-check the active column instead of trying to find them again in the filesystem…

[topherbuckley]

@chschlue and @SembazuruCDE, I don’t think I will be making the same mistake again (fingers crossed), so deleting them is not a solution to the core issue here; new users will use these diodes by default, and use the standard footprints by default, which will lead to an error by default.

[Rene_Poschl]

There is nothing the library team can do for you here. The only option would be to have some form of setting that turns off showing of simulation symbols BUT that would still need to be off by default as otherwise new users who want to use kicad for simulation will not find the symbols.

So in the end as long as you do not understand that “symbol for simulation” means “do not use this symbol if you are not currently simulating” then i doubt anything can be done.

Changing the sort order as you suggest is simply not a solution as the kicad library system is simply way too flexible. I doubt there will be a way to find a sort order specification that would solve your issue while not reducing kicads flexibility.

I mean it is sad that you made a mistake here but to be honest the mistake was that you did not double check if your symbol/footprint pair fits the component you indented to buy. And this is a mistake that will happen at times but it is something where nobody can protect you. If you for some reason would have avoided this specific mistake then you would have learned this lesson later by for example selecting the wrong transistor symbol/footprint pair (SOT-23 is a funny package).

Fully specified symbols* are really the only option to reduce this specific risk. Which is why most symbols in the library are made this way. However, we can simply not have such symbols for every component out there. Especially not for passives as there are simply way too many of them. And we are not going to force anyone to use only fully specified symbols (kicad is flexible by design).

---

*) Fully specified symbols are symbol that have the footprint already assigned named after the set of parts it represents.

[paulvdh]

I agree with Rene here. If you search for a “diode”, then see some 50 search results and blindly pick the first one without thinking why there are so many there is not much others can do to help. Diodes (and LED’s) are known troublesome components. Especially the SMD variants. Sometimes the Anode is marked, and sometimes the Cathode. It is just one of the things everybody who works with electronics has to learn to watch out for because of lack of standardization. SOT-23 is the worst example of this.

But why do the spice libraries have diodes at all?
Why not make the standard device:D and device:D_ALT compatible with spice, which probably means adding alternate pin numbering for spice.
Same for other standard components such as the passives (resistors, capacitors, inductors) These could simply all have idealized spice models attached.

On the other hand it would be a big plus if a bunch of “standard” components was included in the spice libraries. Some opamps such as the NE5532, LM324, TL072. Transistors like BC547 and BC547 etc. I mostly gave up trying to use ngSpice with KiCad after it took me a whole afternoon just to find a model of a NE555 that worked. Trying to get ngSpice to give some valid results is much more work then building the thing on a breadboard.
(I know of the troubles about copyright for spice models, But I think that someone with a lot of knowledge about how those models work, can build a bunch of such models which are good enough to get beginners started with ngSpice in KiCad.)

[topherbuckley]

@Rene_Poschl and @paulvdh,

I’m completely in agreement with you that there are plenty of risks/issues that simply have to be learned from experience (good or bad), but I disagree that this is one of them. Let me reiterate my point I made before as a summary:

As it stands, the default diode symbol is not compatible with many (all?) default footprints, which will lead to a design error not caught by any error checking … by default.

From a UX standpoint do you not see an issue with this? Imagine you were new to the terminal and you just wanted to create a directory with mkdir mydir, would you read through the entire man page for mkdir before executing this command? Not likely. What if, in this theoretical construct, mkdir mydir actually resulted in the deletion of mydir without warning, unless you added some flag mydir -a mydir? Would you not find this counter-intuitive? Sure, you could tell every user who deletes a directory instead of creating one that “you should have read through the man page”, but this is simply bad UX design. If you have the option to make something more intuitive without breaking anything else, I see no reason not to. Just because you know better doesn’t mean everyone else, especially new users, also do.

@Rene_Poschl

Changing the sort order as you suggest is simply not a solution as the kicad library system is simply way too flexible. I doubt there will be a way to find a sort order specification that would solve your issue while not reducing kicads flexibility.

I think you may have missed my most recent comment (Symbol filtering sort order)?

@topherbuckley

What about just renaming the pspice.DIODE to pspice.D and Simulation_SPICE.DIODE to Simulation_SPICE_D similar to how it is done in plenty of the other libraries? That would bump them further down the list without touching the otherwise clever sorting going on there.

I agree that there is no need to change the sorting algorithm, but simply by renaming the spice diodes to D, as occurs with other standard naming conventions, we could stop the spice diodes from being the first (default) choice for users selecting their first diode. I should also point out that the pspice library already calls capacitors C and resistors R, so this isn’t exactly a radical idea.

@Rene_Poschl

So in the end as long as you do not understand that “symbol for simulation” means “do not use this symbol if you are not currently simulating” then i doubt anything can be done.

@paulvdh

If you search for a “diode”, then see some 50 search results and blindly pick the first one without thinking why there are so many there is not much others can do to help.

I have to say, for a forum on an open source project, I’m a bit disappointed by what is coming across as an elitist tone. I’d appreciate if we can focus on the pros and cons of my proposed solution (renaming pspice.DIODE to pspice.D and Simulation_SPICE.DIODE to Simulation_SPICE.D) rather than telling me “you can’t fix stupid”. I have explained, there IS something we can do to help and so far I have read no good reason not to go ahead with such a fix.

Again, I have repeatedly admitted my mistake and owned it. I am merely trying to prevent others from overlooking it in the future, and I believe there is a simple fix for this that would lead to a better UX for KiCAD.

In my defense though, and all others making their first custom component in KiCAD, I will walk you through the process I actually did to show you how easily the things you claim to be obvious can be easily overlooked. And please, for the sake of your new users, put yourself in the shoes of a new user rather than the experienced ones you normally wear. For the diode I made the mistake on, I was trying to create a symbol for this component. Here are the steps I took in alignment with the docs.

1. Opened the symbol editor
2. Searched diode
3. Opened the first few to take a look. psice.DIODE, Simulation_SPICE.DIODE and Device.Laserdiode_1A3C. Didn’t want a laser diode symbol so just chose the first (pspice.DIODE). Note: to the beginner use (me) all I was looking for was a diode symbol to copy. I wasn’t interested in taking other parameters from a specific device, as I planned to assign my own ngspice model, footprint, and 3d model later, so paying further attention as you suggest at this point I don’t think should be expected. To me, all I wanted was the vector graphic of a diode. Also note, that the description here is in size 10 font at the bottom of a fullscreen view (not all users have this window open fullscreen either, making it feasible this isn’t even shown). As a new user, I didn’t even see this description. Details of pivotal importance should not be so small. Not that I am warranting pop-ups, but if I had pressed ok to a pop-up warning about this, then yes, all shame on me and new users, but I dearly hope you can appreciate the ease in which this description can be unintentionally overlooked by those unfamiliar.

4. Exported and saved to a custom library
5. Added the diode to my schematic
6. Edited diode and added Package_TO_SOT_THT.TO-220-2_Horizontal_TabDown
7. Error checked both in eeschema and pcbnew after connecting everything and no errors shown.
8. Built pcb in-house, assembled everything and discovered the diode footprint was opposite polarity.
9. Looked into it further and found the description you mention, which led me to this discussion.

[Rene_Poschl]

@topherbuckley

As it stands, the default diode symbol is not compatible with many (all?) default footprints, which will lead to a design error not caught by any error checking … by default.

The default symbol is in the device lib. Simple as that. The description of the symbol you use CLEARLY says that this symbol is for SIMULATION.

And the simulation symbols do not have footprint filters set. So if you use cvpcb (assign footprint tool) then you will not get any results with the default filter setting. This is an insurance policy to reduce the risk of using these symbols on accident.

Also the symbol even shows the pin numbers at least in the symbol editor. Another way to tell you that this symbol is wrong for the use you intended. Especially when trying to use them as the basis for your custom symbols. Where i would really expect extra checks to be done (why would one make custom symbols if not to double and tribble check the library)

---

And we even have a diode library that holds fully specified symbols.

[topherbuckley]

@Rene_Poschl

The default symbol is in the device lib. Simple as that.

How is this the default? When I search “diode” pspice.DIODE is the first (default) result.

@Rene_Poschl

The description of the symbol you use CLEARLY says that this symbol is for SIMULATION.

Yes. We agree on that. You don’t have to keep restating it.

@Rene_Poschl

And the simulation symbols do not have footprint filters set. So if you use cvpcb (assign footprint tool) then you will not get any results with the default filter setting. This is an insurance policy to reduce the risk of using these symbols on accident.

This is not what I did. See the detailed steps I wrote out in previous post. I followed a standard procedure given in the official documentation.

@Rene_Poschl

Also the symbol even shows the pin numbers at least in the symbol editor. Another way to tell you that this symbol is wrong for the use you intended. Especially when trying to use them as the basis for your custom symbols.

I don’t think you put on the shoes of a new user as I recommended. I know you know to check this; you don’t have to prove your vast knowledge to the rest of us.

My point remains despite your continued efforts to insult my, and other new user’s intelligence:

What pros and cons do you see regarding my proposed solution:

I propose to rename the pspice.DIODE to pspice.D and Simulation_SPICE.DIODE to Simulation_SPICE.D.

Can you please focus your comments on this rather than continued unproductive attacks on other’s intelligence in this open-source community forum please? This solution solves my issue, and my concern for new users. Does this solution pose a problem to you, and if so, objectively how?

[Rene_Poschl]

Changing symbol names breaks old schematics. -> Might become an option in v7 but i am not convinced it is a good idea in v6 as it would make transitioning from v5 a lot harder than necessary.
(with v6 this is less of an issue as the symbols are embedded in the files. which means v7 would be the first place when one can switch to a different name without negatively impacting existing users)

---

However i am no longer the head of the library team. Maybe the new team has shifted the priorities away from keeping the library stable for old users.

---

To make it more clear: I understand that there is an issue here. But i fear there is no simple solution to this issue. Especially not one that can be done so shortly before the v6 release.

[topherbuckley]

Thank you for the objective reasoning, and acknowledging there is an issue. That makes sense, and is a legitimate con to my proposed solution. Other than personal old designs and the 17 example circuits is there anything you are considering when you say:

@Rene_Poschl

it would make transitioning from v5 a lot harder than necessary.

I should mention that I did a grep search on the 17 example circuits and it seems none of them contain any reference to ‘diode’ (case in-sensitve), and the only reference to ‘spice’ is within:

./raspberrypi-gpio-40pin/raspberrypi-gpio-40pin.pro:78:SpiceForceRefPrefix=0
./raspberrypi-gpio-40pin/raspberrypi-gpio-40pin.pro:79:SpiceUseNetNumbers=0

So, this proposed solution should not break anything in them. When trying to open the above mentioned template project I ironically get a lot of missing symbol errors for other symbols though

For the sake of discussion, I tested how this would work by making a simple example circuit based on the pspice.DIODE symbol and lib files:

I then renamed them to D in the .lib and .dcm files, but giving the diode an alias DIODE for the sake of some backwards compatability. So:

Before

pspice.lib

#
# DIODE
#
DEF DIODE D 0 40 Y N 1 F N
F0 "D" 0 150 50 H V C CNN
F1 "D" 0 -175 50 H V C CNN
F2 "" 0 0 50 H I C CNN
F3 "" 0 0 50 H I C CNN
DRAW
P 2 0 1 0 75 100 75 -100 N
P 3 0 1 0 -75 100 -75 -100 75 0 F
X K 1 -200 0 150 R 50 50 1 1 I
X A 2 200 0 150 L 50 50 1 1 I
ENDDRAW
ENDDEF

pspice.dcm

$CMP DIODE
D Diode symbol for simulation only. Pin order incompatible with official kicad footprints
K simulation
F ~
$ENDCMP

After

pspice.lib

#
# DIODE
#
DEF D D 0 40 Y N 1 F N
F0 "D" 0 150 50 H V C CNN
F1 "D" 0 -175 50 H V C CNN
F2 "" 0 0 50 H I C CNN
F3 "" 0 0 50 H I C CNN
ALIAS DIODE
DRAW
P 2 0 1 0 75 100 75 -100 N
P 3 0 1 0 -75 100 -75 -100 75 0 F
X K 1 -200 0 150 R 50 50 1 1 I
X A 2 200 0 150 L 50 50 1 1 I
ENDDRAW
ENDDEF

pspice.dcm

$CMP D
D Diode symbol for simulation only. Pin order incompatible with official kicad footprints
K simulation
F ~
$ENDCMP

This moves it down the sort list significantly as expected:

When reopening the schematic, the resulting error, as expected, is shown below:

You can press ok and it will recover the previous pspice.DIODE from the project cache, no harm done. Although this does “break” old designs, it seems very elementary to revert back to the old lib files by pressing ok here thanks to this recovery functionality.

Alternatively, if I want to reassign the new lib definitions I can press cancel then go to Tools->Edit Symbol Library References:

and replace them all in one go.

So I guess it is now a debate on whether or not this single pop-up and pressing OK is considered more detrimental than the advantages I have spoke of until this point. Unless you see breaking issues that I have not considered here?

[Rene_Poschl]

Any solution needs to be scalable to the full library. While we are now talking about the diode symbol i bet that there are other symbols with similar possible problems. I think this is the main thing you are missing.

As with all things in technology (or life in general) there is no solution. Only tradeoffs. Which side of a tradeoff you fall on highly depends on your personal experience. I personally prefer a stable library as we learned the hard way how bad it gets if we change stuff too drastically (the version 4 to 5 reorg was a nightmare to deal with). You can’t imagine how many angry e-mails, github issues, forum posts, … i personally received for having the audacity to change library names.

[topherbuckley]

@Rene_Poschl

While we are now talking about the diode symbol i bet that there are other symbols with similar possible problems. I think this is the main thing you are missing.

You’re right in that I haven’t been considering the possibility of others, but the inability to fix everything should not be the reason for fixing nothing. Do you see a reason not to fix just the diode as it is a very common base part? I

That’s not to say I wouldn’t be willing to find the remainder. From the limited scope I know of the full library repo, it seems this is only a common issue with the spice libraries (pspice and Simulation_SPICE). Many of these symbols are not directional (0, C, INDUCTOR, R, etc.) , and those that are non-passives, amount to something around 20 symbols. It doesn’t seem to be insurmountable task.

Where do these .lib and .dcm files live on the web btw? I didn’t see them in any of the gitlab or github repos. If you could point me there, I’d be happy to make the proposed changes as a pull request and see what others think as well.

@Rene_Poschl

As with all things in technology (or life in general) there is no solution. Only tradeoffs.

A bit overly pessimistic for my taste in general, but I can definitely see your point regarding this case in particular.

@Rene_Poschl

I personally prefer a stable library as we learned the hard way how bad it gets if we change stuff too drastically (the version 4 to 5 reorg was a nightmare to deal with). You can’t imagine how many angry e-mails, github issues, forum posts, … i personally received for having the audacity to change library names.

I’m doing my best to imagine despite you telling me I can’t . Yes, I can imagine this would be a nightmare to deal with, and sheds a lot of light on why you have been so resistant to this. Thank you for sharing your hard lesson learned.

I could imagine significantly altering names to cause a lot of chaos, but would you forsee such chaos from altering a single component and keeping the old name as an alias? I think keeping old names as an alias is a good effort towards backwards traceability if not compatibility.

I wonder if there would be an easy way to make use of the “recovery” detection that opens when names change such that you can remap names in this same dialog. That would make updating to new versions a lot more streamlined.

[eelik]

There may be other solutions for this problem, like discussed in this issue: https://gitlab.com/kicad/code/kicad/-/issues/6028.

[paulvdh]

A possible improvement may be to add Footprints text to the schematic symbols for the spice libraries with a text something like: “For_Simulation_Only”.

This would alert users that there is an issue when assigning footprints to the simulation symbols, but it may also lead to confusion and users posting questions on the forum.