Semantic problem with KiCad component treatment?

I just want to make clear that I completely understand @robomartin 's proposal and desire for a more rigid semantic system of properties/fields. I don’t think this will be implemented in the short term, and maybe never, but I do understand it.

If we did implement something related to this, it is more likely that we would implement a system allowing someone to define and enforce their own semantics, rather than building a single rigid set of semantics into KiCad itself.

Some of the other points on this thread are known issues that we do plan to improve and/or are already working on, such as:

  • It should be easier to configure BOM exports without needing to write Python code (#9992, #5408, etc)
  • It should be easier to manage library metadata (#11506, #4127, #2141 and many more)
  • It should be easier to implement fully-atomic libraries (#7436, #6941, etc)

I also want to note for those who may not be aware that the KiCad lead developer team does not control the KiCad library project – these are all part of the KiCad project, but a proposal to modify the KiCad standard library content or practices (the KiCad Library Conventions, or KLC) is a question for the library team, not the KiCad developers.

I too am familiar with military / aerospace design requirement. I can’t imagine using Kicad in a integrated system meeting such requirements. I’m thinking you are are talking about some combination PLM and Mfg system.

“on the fly” Back when I was the engineer designing circuits I would start with a schematic (then on 11 x 17 paper). As I roughed out the design I would add resistors and capacitors etc. The detail of these components were not of concern at that time as I was only interested in the basic function. When I thought the design would actually function I would go back and look more closely at the values and components.

As opposed to having a fully functioning tested system and going to a design house to have a PCB made.

BTW my whole career has been in new product design. Most of the challenge has been the unique capabilities of our products. This requires understand the system our product was part of but I’ve thankfully not been that deeply involved is a design phase on your referenced page 20 diagram.

The excerpt you quoted is an example of how overloading a field, like the value field, is a bad idea.

I’m not sure I understand your statement “overloading a field”. Or what it is you are requesting. The value field is just that. It could be ohms, farads or even the base number of an IC. For instance I might use the value field with a base number of an IC. say TLC4040 for a battery charger IC. While this is not a numeric value is it however the general descriptive name of an IC. And it keeps the full part number off the schematic graphic. And if I want to use the value field for some personally cryptic information then that’s my prerogative.

I am all for standardization, having well defined interfaces, and a solid database / library system that can be used by both manufacturers, big shops and individual companies but that has nothing to do with adding default empty fields to all the library symbols.

To me it makes more sense that a company that already has a database (for some other EDA suite) with some set of semantics to keep on using the same semantics for KiCad, instead of juggling between their database semantics and a different set of names for KiCad.
Those company databases are a very valuable asset for them and KiCad should have the flexibility to conform to that instead of mandating some other set of names.

Having a list of preferred semantics for use in KiCad is a good idea to have some guidance for people who want it, but hard coding them into KiCad’s libraries is not a good idea.
It’s also quite easy to write a script to replace one set of keywords to another set in a library, and having a mechanism in place for this is much more practical then hoping the whole world will stick to the same convention. We can’t even agree to use the Metric system. (That’s not really true, almost the whole world uses Metric, except a single significant country and some small area’s).

1 Like

Yes, of course, that’s a normal process. I do that all the time. Most of my designs start as pen-and-paper ideas. This applies to software, electronics and mechanical. I still do state machine and such things on paper before coding. It is often hard to create in the context of a rigid CAD system of any kind, I see it more as a means for documentation once you have a reasonable idea of what it is you are after.

What I will sometimes do on EDA is grab components from the most convenient library and thread together a design. That means I give preference to our vetted components, sometimes without regard for values or details and, if they are not there, I might go for a stock library of some kind just to get moving. The closer I get to the golden component library the less work I create for myself later.

At some point all components have to be from the qualified libraries. If some are missing, this is the moment to do the work require to add them.

Once you do that and make all appropriate replacements it’s one turn of the crank to get a BOM you can review and order and off to layout it is.

I would guess everyone has a process along these lines to one degree or another. You have to get your thoughts down first before you can engage in detail work.

Ah, good point. I am using language from object-oriented programming without realizing a lot of people might have no clue what I am talking about. Thanks for pointing this out.

In case you happen to be interested in that context, Wikipedia does a decent job:

https://en.wikipedia.org/wiki/Function_overloading

In this context I use it to mean the use of a field for more than one thing and that these things are not necessarily the same.

Let me use a completely unrelated example:

You go to the doctor. They make you fill out one of those horrid forms nobody likes to fill out. One of the items has a label that reads “height”. You write in your height and move on.

A few minutes later the assistant comes over and says, “You didn’t enter your weight. We need that.”.

You look at the form. There is no field tagged “weight”.

You ask. They say "Oh, just put it in the “height” field.

What do I do with my height?

Oh, just cross it out.

Do you need it?

Yeah, but there’s no place to put it. Just write it somewhere on the side of the page.

So…you want me to put my weight where it says “height” and my height on the side of the page?

Yes.

OK.

The reuse of the “height” field to record the weight is, in programming terms, akin to function overloading. It isn’t a perfect parallel but most programmers would understand what I meant when I used the term. Sorry for not realizing I was speaking gibberish to everyone else.

Anyone can understand the obvious problems with such a medical form. Imagine if every office used the “height” field differently. Imagine if every office had a different idea on where to record the weight and what to call it. It would be a mess and your medical records could not be moved shared from doctor to doctor without a ton of work.

That’s the issue I am presenting here. The term “semantics”, as I apply it here, is about having fields that capture the appropriate meaning. The “value” field should never have a part number or be linked to the symbol name. Anyone can understand that the “footprint” field should not have the URL to the datasheet and that the “datasheet” field should not have a link to the 3D model. Everyone gets this and would likely agree with this. Well, it isn’t any different with the “value” field.

From this it follows that we need two standard fields: “part number” and “manufacturer”, along with a “value” field that has no connection to the symbol name.

This isn’t about forcing people into workflows as much as it is about creating a sensible starting point from which a greater ecosystem can evolve. Nobody in the electronics industry orders part by hand-waving. As a famous race car builder and mechanic used to say “When all the smoke and bullshit clears out, you have to drive the car and win the race”. Well, when all the huffing and puffing clears out, you have to have a real orderable part number and the manufacturer’s name to be able to get anywhere with anything other than hobby electronics or just hacking around. They are, without a doubt, absolutely necessary. And KiCad does not provide official support for any form of this in a standardize manner that would enable the user and software author community to expand capabilities beyond what anyone could possibly imagine.

I think I made this clear but I’ll try to clarify further here in case I haven’t.

I am not proposing anything rigid be put into place. Flexibility is key or you won’t be able to address the myriad needs out there.

What I am suggesting is that a starting point that addresses the deficiencies I have highlighted would be important in order to greatly enhance usability and enable an ecosystem of libraries and tools to grow around this baseline.

Advanced users should be able to completely ignore this and do as they wish. No question there at all. I think it is precisely the newbies, students and intermediate users who would greatly benefit from addressing these issues in a sensible and flexible manner. As students become professionals they might very well adopt different methodologies and adapt to whatever it is their company likes to do. Nothing in what I am proposing challenges or negates this level of flexibility.

The kinds of changes I am discussing require matching functionality at the application layer. I would imagine the library team has nothing to do with this. Right? And so, if the library team decides to add fields that require intelligence within the application, unless this work happens, the addition might not have the desired effects.

As you have dscovered it only adds the field for the symbols where you fill in the field. If you were populating the schematic from an enhanced DB where parts have IDs then the field would be filled in.

Users should not push the contents of the part field back to the library, that’s the job of the librarian.

1 Like

Or you could use a configurable BOM generator like KiBOM which can adapt to the actual names of the fields and output the CSV with different heading names. You’ll need a flexible BOM tool to handle the different column names and orders required by various assembly houses anyway. It’s also likely that such a BOM tool will have to do more lookups on the DB, to get the MPN for example. So you’ll have to base your argument for a standard name on other reasons.

I agree there should be a part ID field. I doubt if the developers will want to standardise on the name. In the absence of that, you can just publish your software and by being the first establish your chosen name by popularity. It doesn’t have to be a progress blocker, just pick a reasonable name and if the developers agree to a standard name later, just do a global edit.

I agree that there should be standardized field names for Part_Source or MFR and Part_Number or PN or something similar. They should be left blank in the standard libraries, but having standard names allows the software to do things to support the use of a part number, be it a manufacturer’s part number or a house part number.

There are issues with the proposed alternate of just adding these fields to your own custom libraries. For example, I copied the L272 dual op-amp from the standard library into a personal global library and edited the symbol to add the two new fields. Then I placed all three sub units on schematic and assigned them to U9. I set the part source and part number fields on the first unit, U9A, but those values were not displayed when editing the second unit. I could then add different part source and part number values to these fields for the other units, U9B and U9C. This leads to the three sub-units of a single dual op-amp part showing different values for the part source and part number as shown in the screen shot below. (Note, the TI and onsemi part numbers are fictitious).

This is clearly not right. The software should know that the three units in a part can only have one part number. It could display the part number values for only the first unit or for all units, and allow editing from the first unit only or from any unit. If it does the later, it should update the one and only part number for the part and update the displayed part numbers for all units. Editing the part number for unit B should change the value displayed for unit A and C.

The current situation arises because the software does not attach any semantic meaning to the part number added by the user. It is simply a string that can be displayed beside the part’s symbol. Right now it doesn’t know that all the units in a part must come from the same part with the same part number. This could be enforced by KiCad if the part number were always there (even if blank) in a field with a known name.

I’m sure that other uses for part number fields would become apparent if they were always available.

I think the KLC could be amended to add recommendations for library maintainers to use these fields correctly to build consistency between third party libraries.

It’s a simple change to the KiCad standard libraries which would encourage uniform use without undue burden on the library maintainers. The changes to the KiCad software could be made as time allows since they are not needed for the new fields to be used, but might add new features or checks in the future. No one would be forced to use the new fields.

I don’t think this proves that anything is wrong with user defined fields because you started from the library end. If you start from the schematic editor end the results are different.

I took one of my designs and added the field LCSC to it. Next I went into Tools > Edit Symbol Fields and added a random part number C244. When I edit the properties of the (unit of) the IC in question, there is only one LCSC field no matter which unit you choose. (Similar to the datasheet field.)

A correct implementation would populate the part number from a database, not the library. So another type of library would be needed. There are already KiCad libraries and KiCad legacy libraries. It might also be necessary to tweak the application to know about read-only fields.

If anything this shows that the stock libraries should not be modified. If anything the additional fields should come from a full-fledged database.

For the life of me I can’t understand this obsession on a name. So long as it isn’t “popcorn” to actually mean “part number”, frankly, nobody should care. There are probably a dozen viable field names that convey the correct meaning. I don’t care. Nobody should, so long as the semantic meaning is reasonably close to one of two possible ideas: manufacturer part number, internal part number or something like unique part identifier. The first one requires a manufacturer field as an add-on for disambiguation (same exact part number made by multiple companies), the other two are -generally speaking-- only useful with an external database.

Because my argument is in favor of a better out of the box experience and functionality without having to resort to external tools, “part number” and “manufacturer” fields are required, along with a “value” field that is fully independent of the symbol name.

I have explained this multiple times now and, for some reason, people keep arguing something I am not saying at all. If you have the following fields:

  • part number
  • manufacturer
  • value (independent from symbol name)

You can ignore all of them and choose one for your internal part number or database ID. That, when combined with suitable API functions for these fields might provide most of what people might need to work with database-managed parts or some other scheme.

What it does do is create a far better out of the box starting point for everyone who will not use a database. Judging from my experience in the Altium Designer community, I would not be surprised if 80% to 90% of KiCad users will never touch a database. Which means this out of the box field definition and the ecosystem it enables would be nothing less than fantastic for them.

Heck one could even add a fourth field, call it “db link” and give it special properties in code. Blank by default. Not a problem. Storage is dirt cheap. Having a few semantically useful blank fields is nothing.

As an experiment, I just took the stock “Connector_Generic.kicad_sym” library with 274 components and used Notepad++ to add the following fields: “_part_number”, “_manufacturer”, “_value”, “_db_link”. Using an underscore to distinguish from existing fields. They are all set to empty by default (well, “~” or they won’t show-up).

This took a 3.8 MB file and added 100K to it. In other words: nothing, a rounding error. One could easily automate this to process every single stock library.

The current KiCad experience would not change whatsoever. However, if these fields are there in the library, for every component, as a installation default, people could do very useful things with them. Or, they could simply ignore them all and keep working with the four mandatory fields as they currently exist.

If the Python API could be used from the schematic editor and had a few useful tools related to these fields, that would be all anyone might need to implement a full DB driven workflow or a range of modalities and enhancements that do not require an external database at all. All it requires is the KiCad team (whichever would have to make the change) deciding to issue an out of the box standard that would vastly benefit everyone involved.

At a first approximation this isn’t that difficult at all. It won’t provide all the functionality I envision, but it would for sure improve the experience.

I am thinking I might test the idea an publish a full replacement for the KiCad libraries with these fields along with plugins or some basic code that can make use of them and guidance on usage beyond that. That could be a very interesting experiment. I am going to seriously think about it.

  (symbol "Conn_01x01" (pin_names (offset 1.016) hide) (in_bom yes) (on_board yes)
    (property "Reference" "J" (id 0) (at 0 2.54 0)
      (effects (font (size 1.27 1.27)))
    )
    (property "Value" "Conn_01x01" (id 1) (at 0 -2.54 0)
      (effects (font (size 1.27 1.27)))
    )
    (property "Footprint" "" (id 2) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "Datasheet" "~" (id 3) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_part_number" "~" (id 7) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_manufacturer" "~" (id 8) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_value" "~" (id 9) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "_db_link" "~" (id 10) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_keywords" "connector" (id 4) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_description" "Generic connector, single row, 01x01, script generated (kicad-library-utils/schlib/autogen/connector/)" (id 5) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (property "ki_fp_filters" "Connector*:*_1x??_*" (id 6) (at 0 0 0)
      (effects (font (size 1.27 1.27)) hide)
    )
    (symbol "Conn_01x01_1_1"
      (rectangle (start -1.27 0.127) (end 0 -0.127)
        (stroke (width 0.1524) (type default) (color 0 0 0 0))
        (fill (type none))
      )
      (rectangle (start -1.27 1.27) (end 1.27 -1.27)
        (stroke (width 0.254) (type default) (color 0 0 0 0))
        (fill (type background))
      )
      (pin passive line (at -5.08 0 0) (length 3.81)
        (name "Pin_1" (effects (font (size 1.27 1.27))))
        (number "1" (effects (font (size 1.27 1.27))))
      )
    )
  )

I think you are better off going with an external database from the start rather than futzing with the stock libraries. If you look at the post above, you’ll see a drawback another user discovered. Then you can settle on one unique part number field that is used to reference everything known about that part, and you won’t have two fields in the key. This requires KiCad to know about a different type of library though.

Ok, you don’t care about the name. But you propose to modify the stock libraries. I think a better way to go is to create your own DB implemtation of the libraries with the extra fields you want.

You are very fluent at writing. There’s a drawback to writing too much though, your key points get lost. I’m not the only one trying to extract the salient details from the long passages.

1 Like

Thank you for running this experiment. I’ll reproduce it when I have the opportunity. I have been pounding on the table asserting that semantics is important. Your experiment provides further proof for this assertion.

The more I think about this and explore I am starting to think this is a seriously broken issue that could qualify for a top priority.

The overloading (multi-use) of the value field and its linkage to the symbol name and the lack of semantically useful places for a part number and the manufacturer are, in my view, serious issues. All the current approach does is truly make a mess of the out of the box experience and make a mess of doing something as simple as creating a usable BOM without plugins that further lock you into a semantically broken structure.

As I have evolved this idea I am now thinking the fields should be:

  • part number
  • manufacturer
  • value (independent from symbol name)
  • db link

If the part number is filled, a right click on the part in schematic or pcb should launch a script that is initially configured to run a simple google search using the part number and manufacturer fields. The user could modify this script to do anything else that might be useful with access to the part number and manufacturer. For example, ignore the manufacturer and use the part number as an internal/house number to do whatever they need to do.

The same would be true of “db link”. If filled, this field could trigger an external script the user can do anything they wish. Maybe it’s “db key” or whatever. You enter your internal part number here and ignore the other fields entirely. Right click and you get to pass that number to your DB and do as you wish.

This, in my opinion, would be a vast improvement from what currently happens with KiCad if you don’t write code or load it up with plugins…and there’s still no place to put down a real part number, internal or external.

No, because 80 to 90% of users have no need for an external database. This would definitely complicate their lives. The out of the box experience cannot be a db-driven system. This would add an order of magnitude more work to everyone who volunteers their time and effort to develop and support KiCad.

Keep it simple. Add a few reasonable fields to improve semantics, the experience, ecosystem and functionality. Those who want or need a DB can use one or more of the provided fields and a little enhancement to the API’s to do anything they want.

Sometimes details matter. And sometimes it takes a significant written effort to convey a message effectively. One of the things I have found surprising in this thread is how much people read into things that are simply not being said. So I have to repeat myself and modify the message here and there hoping it will finally click. And then there’s the issue of terminology. The simple example being me using the term “overloading” without realizing it actually meant nothing to anyone without the relevant CS background. Little things like that. Communication can suck sometimes.

Users who don’t need external databases don’t have to use it, they can continue to use the simple stock implementation. A DB based symbol library would be a different type of library that would be registered in the symbol library table. It would be coupled to the DB with a ODBC shim. This would be provided by the user.

Edit: you do know that KiCad supports at least 2 library formats, KiCad and KiCad legacy (< v6), right? So another type of library, a DB based one, could be supported.

I think in trying to augment the stock libraries you are just straining its capabilities, will hit limitations and will have to pile on kludges. Not to mention that the KiCad librarian(s) will not want to do this work.

But not all at once. You can zoom in as required.

Not sure why you keep discussing DB based libraries. Not relevant in the sense that what I am suggesting needs fixing has nothing whatsoever to do with them. The only connection is that the approach I am suggesting would actually facilitate working with an external database while providing a significant enhancement to stock out-of-the-box KiCad which is what my proposal would target.

Respectfully, please stop going on about database libraries. Those of us who use them will continue to do so in multiple ways. The vast majority of users in any EDA environment do not use database libraries. There are many reasons for this. I would say the main one is that this ads a level of complexity that most individual users and small teams just don’t want to deal with. Last I checked Altium Designer has an installed base of around 50K users. I guarantee you that over 80% of them, if not over 90% have never touched and will never touch a database-driven library process.

I have founded and operated three technology companies so far in my life. We have never used DB driven libraries with Altium or any other EDA tool. The only places I’ve used DB’s has been when I worked at large aerospace companies who had entire teams of librarians and IT to maintain the requisite system, including having teams of engineers dedicated to qualifying the parts, etc. That said, for various reasons, we are starting to lean towards finally making the transition, maybe next year.

So, please, if you want to continue to discuss my proposal, please understand that going on about database libraries simply isn’t relevant at all except for the positive side effect that the 5% to 10% of users for whom this might be important will likely have a very simple path to make that happen. Nobody else cares. Because they don’t need it.

Because I think you’re handicapping yourself working with KiCad’s text based libraries. But it’s your project.

There’s nothing wrong with KiCad’s s-expression based library format. I worked with LISP on and off for probably ten years. I also wrote some pretty large plug-ins for AutoCAD back in the day using AutoLISP (their flavor of LISP). It’s great. It works. Not sure JSON would be better. I think it was a good decision to use s-expressions for libraries, schematic and pcb (not sure where else the syntax is used).

Talking about standards, this is one standard that allows a great deal of creativity in manipulating data structures in KiCad. Good stuff.

It’s not the format. Do a mental walkthrough of how a user would set up a modifiable library that can have the additional fields starting from the distributed library packages. Then the multiple users on a machine scenario, including locking for updates. Then think about how to handle updates from the upstream library packages. Not so OOtB experience anymore. I’ll leave the exercise to you.

Not relevant for >90% of users. Adding the fields to the libraries is super simple. All you have to do is insert one s-expression per component. Super simple code. You could probably process all libraries in the distribution in less than five minutes. You’d end-up with all the right fields in every symbol and they would all be set to be blank (well, a “~” squiggle, otherwise KiCad won’t display them). Done deal.

Seriously, this part isn’t hard. The part that isn’t that simple is to get KiCad to play nice with the fields.
Despite not having a Python API for Schematic I have some ideas for that as well that might work very well. I’ll have some time to experiment with this in about two months. We have a lot of product to manufacture and ship in the next ten weeks or so, all my coding time is consumed by the embedded and system code I have to write for our shipping products. I need to learn more about KiCad internals anyway. That I can do as a low priority task going forward.

You haven’t addressed the writable library and update from stock packages issues. You’re basically saying to users: to use these fields then make a copy of all the stock libraries and do these edits, and if the stock libraries get updated, you’ll have to transfer your part ids, or forgo the updates. Otherwise just stick to using private libraries. And by the way if you want to share the library between multiple librarians, then they must coordinate so that they don’t update at the same time.

It’s nothing more than users can already do given a HOWTO, perhaps scripts could automate parts of it. Such a disappointment after all that talk.