Semantic problem with KiCad component treatment?

Software
128

I don’t think this proves that anything is wrong with user defined fields because you started from the library end. If you start from the schematic editor end the results are different.

I took one of my designs and added the field LCSC to it. Next I went into Tools > Edit Symbol Fields and added a random part number C244. When I edit the properties of the (unit of) the IC in question, there is only one LCSC field no matter which unit you choose. (Similar to the datasheet field.)

symbol-properties
symbol-properties841×528 38.5 KB

A correct implementation would populate the part number from a database, not the library. So another type of library would be needed. There are already KiCad libraries and KiCad legacy libraries. It might also be necessary to tweak the application to know about read-only fields.

If anything this shows that the stock libraries should not be modified. If anything the additional fields should come from a full-fledged database.

129

For the life of me I can’t understand this obsession on a name. So long as it isn’t “popcorn” to actually mean “part number”, frankly, nobody should care. There are probably a dozen viable field names that convey the correct meaning. I don’t care. Nobody should, so long as the semantic meaning is reasonably close to one of two possible ideas: manufacturer part number, internal part number or something like unique part identifier. The first one requires a manufacturer field as an add-on for disambiguation (same exact part number made by multiple companies), the other two are -generally speaking-- only useful with an external database.

Because my argument is in favor of a better out of the box experience and functionality without having to resort to external tools, “part number” and “manufacturer” fields are required, along with a “value” field that is fully independent of the symbol name.

I have explained this multiple times now and, for some reason, people keep arguing something I am not saying at all. If you have the following fields:

  • part number
  • manufacturer
  • value (independent from symbol name)

You can ignore all of them and choose one for your internal part number or database ID. That, when combined with suitable API functions for these fields might provide most of what people might need to work with database-managed parts or some other scheme.

What it does do is create a far better out of the box starting point for everyone who will not use a database. Judging from my experience in the Altium Designer community, I would not be surprised if 80% to 90% of KiCad users will never touch a database. Which means this out of the box field definition and the ecosystem it enables would be nothing less than fantastic for them.

Heck one could even add a fourth field, call it “db link” and give it special properties in code. Blank by default. Not a problem. Storage is dirt cheap. Having a few semantically useful blank fields is nothing.

As an experiment, I just took the stock “Connector_Generic.kicad_sym” library with 274 components and used Notepad++ to add the following fields: “_part_number”, “_manufacturer”, “_value”, “_db_link”. Using an underscore to distinguish from existing fields. They are all set to empty by default (well, “~” or they won’t show-up).

This took a 3.8 MB file and added 100K to it. In other words: nothing, a rounding error. One could easily automate this to process every single stock library.

The current KiCad experience would not change whatsoever. However, if these fields are there in the library, for every component, as a installation default, people could do very useful things with them. Or, they could simply ignore them all and keep working with the four mandatory fields as they currently exist.

If the Python API could be used from the schematic editor and had a few useful tools related to these fields, that would be all anyone might need to implement a full DB driven workflow or a range of modalities and enhancements that do not require an external database at all. All it requires is the KiCad team (whichever would have to make the change) deciding to issue an out of the box standard that would vastly benefit everyone involved.

At a first approximation this isn’t that difficult at all. It won’t provide all the functionality I envision, but it would for sure improve the experience.

I am thinking I might test the idea an publish a full replacement for the KiCad libraries with these fields along with plugins or some basic code that can make use of them and guidance on usage beyond that. That could be a very interesting experiment. I am going to seriously think about it.

  (symbol "Conn_01x01" (pin_names (offset 1.016) hide) (in_bom yes) (on_board yes)
    (property "Reference" "J" (id 0) (at 0 2.54 0)
      (effects (font (size 1.27 1.27)))
    )
    (property "Value" "Conn_01x01" (id 1) (at 0 -2.54 0)
      (effects (font (size 1.27 1.27)))
    )
    (property "Footprint" "" (id 2) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "Datasheet" "~" (id 3) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_part_number" "~" (id 7) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_manufacturer" "~" (id 8) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_value" "~" (id 9) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_db_link" "~" (id 10) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_keywords" "connector" (id 4) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_description" "Generic connector, single row, 01x01, script generated (kicad-library-utils/schlib/autogen/connector/)" (id 5) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_fp_filters" "Connector*:*_1x??_*" (id 6) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (symbol "Conn_01x01_1_1"
      (rectangle (start -1.27 0.127) (end 0 -0.127)
        (stroke (width 0.1524) (type default) (color 0 0 0 0))
        (fill (type none))
      )
      (rectangle (start -1.27 1.27) (end 1.27 -1.27)
        (stroke (width 0.254) (type default) (color 0 0 0 0))
        (fill (type background))
      )
      (pin passive line (at -5.08 0 0) (length 3.81)
        (name "Pin_1" (effects (font (size 1.27 1.27))))
        (number "1" (effects (font (size 1.27 1.27))))
      )
    )
  )
130

I think you are better off going with an external database from the start rather than futzing with the stock libraries. If you look at the post above, you’ll see a drawback another user discovered. Then you can settle on one unique part number field that is used to reference everything known about that part, and you won’t have two fields in the key. This requires KiCad to know about a different type of library though.

Ok, you don’t care about the name. But you propose to modify the stock libraries. I think a better way to go is to create your own DB implemtation of the libraries with the extra fields you want.

You are very fluent at writing. There’s a drawback to writing too much though, your key points get lost. I’m not the only one trying to extract the salient details from the long passages.

1 Like
131

Thank you for running this experiment. I’ll reproduce it when I have the opportunity. I have been pounding on the table asserting that semantics is important. Your experiment provides further proof for this assertion.

The more I think about this and explore I am starting to think this is a seriously broken issue that could qualify for a top priority.

The overloading (multi-use) of the value field and its linkage to the symbol name and the lack of semantically useful places for a part number and the manufacturer are, in my view, serious issues. All the current approach does is truly make a mess of the out of the box experience and make a mess of doing something as simple as creating a usable BOM without plugins that further lock you into a semantically broken structure.

As I have evolved this idea I am now thinking the fields should be:

  • part number
  • manufacturer
  • value (independent from symbol name)
  • db link

If the part number is filled, a right click on the part in schematic or pcb should launch a script that is initially configured to run a simple google search using the part number and manufacturer fields. The user could modify this script to do anything else that might be useful with access to the part number and manufacturer. For example, ignore the manufacturer and use the part number as an internal/house number to do whatever they need to do.

The same would be true of “db link”. If filled, this field could trigger an external script the user can do anything they wish. Maybe it’s “db key” or whatever. You enter your internal part number here and ignore the other fields entirely. Right click and you get to pass that number to your DB and do as you wish.

This, in my opinion, would be a vast improvement from what currently happens with KiCad if you don’t write code or load it up with plugins…and there’s still no place to put down a real part number, internal or external.

132

No, because 80 to 90% of users have no need for an external database. This would definitely complicate their lives. The out of the box experience cannot be a db-driven system. This would add an order of magnitude more work to everyone who volunteers their time and effort to develop and support KiCad.

Keep it simple. Add a few reasonable fields to improve semantics, the experience, ecosystem and functionality. Those who want or need a DB can use one or more of the provided fields and a little enhancement to the API’s to do anything they want.

Sometimes details matter. And sometimes it takes a significant written effort to convey a message effectively. One of the things I have found surprising in this thread is how much people read into things that are simply not being said. So I have to repeat myself and modify the message here and there hoping it will finally click. And then there’s the issue of terminology. The simple example being me using the term “overloading” without realizing it actually meant nothing to anyone without the relevant CS background. Little things like that. Communication can suck sometimes.

133

Users who don’t need external databases don’t have to use it, they can continue to use the simple stock implementation. A DB based symbol library would be a different type of library that would be registered in the symbol library table. It would be coupled to the DB with a ODBC shim. This would be provided by the user.

Edit: you do know that KiCad supports at least 2 library formats, KiCad and KiCad legacy (< v6), right? So another type of library, a DB based one, could be supported.

library-types
library-types1000×500 54.7 KB

I think in trying to augment the stock libraries you are just straining its capabilities, will hit limitations and will have to pile on kludges. Not to mention that the KiCad librarian(s) will not want to do this work.

But not all at once. You can zoom in as required.

134

Not sure why you keep discussing DB based libraries. Not relevant in the sense that what I am suggesting needs fixing has nothing whatsoever to do with them. The only connection is that the approach I am suggesting would actually facilitate working with an external database while providing a significant enhancement to stock out-of-the-box KiCad which is what my proposal would target.

Respectfully, please stop going on about database libraries. Those of us who use them will continue to do so in multiple ways. The vast majority of users in any EDA environment do not use database libraries. There are many reasons for this. I would say the main one is that this ads a level of complexity that most individual users and small teams just don’t want to deal with. Last I checked Altium Designer has an installed base of around 50K users. I guarantee you that over 80% of them, if not over 90% have never touched and will never touch a database-driven library process.

I have founded and operated three technology companies so far in my life. We have never used DB driven libraries with Altium or any other EDA tool. The only places I’ve used DB’s has been when I worked at large aerospace companies who had entire teams of librarians and IT to maintain the requisite system, including having teams of engineers dedicated to qualifying the parts, etc. That said, for various reasons, we are starting to lean towards finally making the transition, maybe next year.

So, please, if you want to continue to discuss my proposal, please understand that going on about database libraries simply isn’t relevant at all except for the positive side effect that the 5% to 10% of users for whom this might be important will likely have a very simple path to make that happen. Nobody else cares. Because they don’t need it.

135

Because I think you’re handicapping yourself working with KiCad’s text based libraries. But it’s your project.

136

There’s nothing wrong with KiCad’s s-expression based library format. I worked with LISP on and off for probably ten years. I also wrote some pretty large plug-ins for AutoCAD back in the day using AutoLISP (their flavor of LISP). It’s great. It works. Not sure JSON would be better. I think it was a good decision to use s-expressions for libraries, schematic and pcb (not sure where else the syntax is used).

Talking about standards, this is one standard that allows a great deal of creativity in manipulating data structures in KiCad. Good stuff.

137

It’s not the format. Do a mental walkthrough of how a user would set up a modifiable library that can have the additional fields starting from the distributed library packages. Then the multiple users on a machine scenario, including locking for updates. Then think about how to handle updates from the upstream library packages. Not so OOtB experience anymore. I’ll leave the exercise to you.

138

Not relevant for >90% of users. Adding the fields to the libraries is super simple. All you have to do is insert one s-expression per component. Super simple code. You could probably process all libraries in the distribution in less than five minutes. You’d end-up with all the right fields in every symbol and they would all be set to be blank (well, a “~” squiggle, otherwise KiCad won’t display them). Done deal.

Seriously, this part isn’t hard. The part that isn’t that simple is to get KiCad to play nice with the fields.
Despite not having a Python API for Schematic I have some ideas for that as well that might work very well. I’ll have some time to experiment with this in about two months. We have a lot of product to manufacture and ship in the next ten weeks or so, all my coding time is consumed by the embedded and system code I have to write for our shipping products. I need to learn more about KiCad internals anyway. That I can do as a low priority task going forward.

139

You haven’t addressed the writable library and update from stock packages issues. You’re basically saying to users: to use these fields then make a copy of all the stock libraries and do these edits, and if the stock libraries get updated, you’ll have to transfer your part ids, or forgo the updates. Otherwise just stick to using private libraries. And by the way if you want to share the library between multiple librarians, then they must coordinate so that they don’t update at the same time.

It’s nothing more than users can already do given a HOWTO, perhaps scripts could automate parts of it. Such a disappointment after all that talk.

140

You are confusing experimentation with an actual solution. This is not a solution by any stretch of the imagination, just a test, one of many to come.

141

Well, all the best then. Really.

142

So what’s the workflow that you have in mind? Say these two (or three) fields are added to all standard libraries, but left empty. The standard libraries are write protected, so the user is left with two alternatives:

  • Add the symbol to the schematic.
  • Edit the symbol properties and fill in the fields.

The drawback to this is that if the same symbol is added again, the values are not filled in in that instance. The symbol can however by copied and pasted in the schematic to include the data.

OR

  • Copy the symbol to a personal library.
  • Fill in the fields.
  • Add the symbol to the library.

Which of these two scenarios do you have in mind, or both? Just to make sure you have the same picture.

143

weird argumentation as the standardization of the field name is also not relevant for 90% of the users.

Its ok that you don’t understand why some want a special name for their part number field but this does not mean that they are wrong. imagine for example they use additional programs and scripts which work exactly with these defined field names and adjusting the software accordingly would be a painful or even not possible process.

you also keep ignoring the fact, that keeping this field empty (as you recommended if not used) can cause confusion in different levels of the production chain → far from ideal and time consuming.

144

@dennischi wrote: an example about different field-values for a multi-unit-symbol where all fields should have the same value. … This is clearly not right… The current situation arises because the software does not attach any semantic meaning to the part number added by the user.

two point from this quotation:

  • “This is clearly not right” —> agree.
  • but the conclusion is false. This situation is not caused by the use of user-setable fields. This is a simple bug. You can provoke this situation also with the much hyped “first class mandatory fields”. I can create the same situation with the value/footprint-fields. So there is no difference in creating this discrepancy for “first class mandatory fields” and user-setable “second class custom fields”. → this example is no argument for introducing additional mandatory fields, there are better/other arguments needed
  • to help you with this bug and circumvent this issue: use the symbol properties dialog (or the symbol field table dialog) to enter the new value for mandatory (value/footprint) and custom fields. Than the string propagates to all subunits. Only the “Edit custom field” dialog allows different strings for every subunit.
  • to help with fixing this bug: you could prepare a nice demonstration-schematic. With an original multi-unit-symbol with uniform custom fields. And a copy of this multi-unit-symbol (as your screenshot above) with different field-strings for every subunit. Together with a description how you created this situation. Together with a step-by-step-description uploading to gitlab - wait until fix is ready.
145

I have reproduced this bug, reported it on gitlab and added both a link to your post and an example project.

If you have further comments, please add them to:

146

I did a bit of research and made a new thread with a similar but more flexible proposal in:

And as I have also added as a header to that thread, I discovered it is already implemented in KiCad.
bangheadonwall

The only thing missing is the semantics for some of those "pre defined names in the template.

147

Your example is flawed because it is quite possible for U1, U2, and U3 to be units of three independent parts that just happen to be placed close together on the schematic. It is only a problem if units with the same base designator, Unnn, have different part numbers.