Semantic problem with KiCad component treatment?

As your contributions seem to repeat again and again I will take only two new aspects from the last answer:

Let’s say, for example, that Yageo was interested in providing KiCad libraries for their entire product line. Where do they place their part numbers? Where do they store the values? The value field in the libraries is the same as the symbol name? How can Yageo --or anyone else for that matter-- delivery usable libraries that would enrich the KiCad ecosystem and experience.

I know it’s impossible to believe but somehow (I don’t know how they managed that, maybe magic) for instance digikey has build a digikey-parts library for kicad. Idon’t know how good it is and how regularly maintained it is, but a quick look (took me 2 minutes - less than writing this text) showed they managed to get a digikey-order-number, the manufacturer, MPN, and many other things in. You are right, it was impossible.

And, BTW, if schematic does not yet have Python API access none of this can be done by users like me without forking KiCad and bifurcating the ecosystem.

As I wrote earlier (not in this thread): also the kicad developers deserve some time to sleep. Python API access for schematic is on the roadmap. Hopefully it will get into v7 (I’m waiting eagerly for this). If not possible, it will get into v8. Or v9. (timeline: it’s ready if it’s ready).
This is a really needed feature for corporate usage - at this point you get my full support.

We read feedback and suggestions from users, and definitely care. What we don’t do is jump to implementing every suggestion from every user. Part of product management is figuring out how to average out user sentiment into a set of decisions that optimizes for developer availability, the proportions of users who care about a specific things, and what workarounds exist for feature requests (and so on). Unlike with proprietary software companies, we try to be as open as possible about our decision-making processes in this regard, even when the resulting decisions don’t make every user happy.

5 Likes

Oh yes! Much more often then I’d like.

Good question! I have no idea how they’ve chosen to do this. Have a look at the available libraries and see how they’ve chosen to do this. The libraries I know of are:
site to download footprint, schematic, 3D
snapeda
ultralibrarian
digikey
There’s also a few libraries available in the Plugin and Content Manager.

This I agree on. The way it currently works is not optimal for atomic libraries. It works, but if one wants a field holding the component value one must create a new field instead of using the Value field. Setting the Name field to the value isn’t an option IMHO.

1 Like

It would help if you did so yourself.
I (and others) have mentioned a number of times that adding empty field to all the (currenty 17000+) schematic symbols is useless, and you just keep repeating your own arguments without giving any comment on that.

I’ve also given the idea of defining those semantic names, and adding them by automated tools when needed, and you have also chosen to ignore that. You give no feedback over other peoples ideas.

Yageo could easily add a “Yageo_Part_Number” field if they want to release such a library. It’s not ideal, but in the end it does not matter much. Scanning several fields is just a few lines of script.
… And if default names are defined, then with a few lines of script can modify such a database.

If Yageo wanted to create such a library now, a much bigger problem is that they would have to copy the graphics of the resistor and capacitor schematic symbols into all their parts because of limitations in the KiCad libraries. As long as those underlying limitations are not resolved it can not be done “properly”.

The current BOM scripts are a mess, I give you that, but just adding those fields does not change much.

The fact that you are still reading a thread that is over 100 posts backs up your point. I don’t know how many developers read and don’t post but enough actively participate that I have to believe the input here helps guide (not dictate) the development process. I don’t have any idea of the total user base and how their needs break down. I suspect most users don’t experience enough problems that they feel the need to even register an account here.

As a software developer myself, I understand exactly what you are saying and appreciate the attention and open communications. In fact, I opted to run this thread here rather than to file a request precisely not to waste your time with a half-assed idea that was not challenged or vetted in any way. In other words, I wanted push-back and opinions before deciding to potentially waste your time with a request.

This thread, as uncomfortable as it has gotten here and there (I was forced to ignore one person, everyone else has been great), has served my purpose, which was to put this through some kind of a first order filter, check my assumptions and, yes, despite what some might think, be challenged. I don’t know any good engineer who does not welcome being challenged.

The conversation has also solidified some of my thinking on this topic and changed some of it as well. It led me to understand some of the extremes. It is also forcing me to get deeper into KiCad before formulating a final proposal I would be willing to submit for consideration.

And, no, of course I don’t expect anyone to jump on this. That would be a silly assumption to make. If it gets done it will get done when it gets done. I am sure the team is starving for time and resources and what you are doing is nothing less than amazing. I don’t have the time right now, but, when I do, and once I get deeper into the codebase, it is my full intent to help in the effort to the extent I might be able to. In other words, I am looking to hopefully become a valuable contributor rather than just wasting clock cycles.

The next step for me is to dive deeper into what can and cannot be done through Python today. I have done a little bit of very light work in this domain, one of which I published here:

https://forum.kicad.info/t/is-there-a-fast-way-to-connect-a-copper-pour-to-a-net/34642/8

I have a whole list of plugins I want to work on next.

Thanks for your support and helpful feedback on everything.

Could I bother you to have a look at this? I am trying to understand the underlying mechanism:

https://forum.kicad.info/t/multiple-clones-of-r-us-but-not-c/35529

Thanks.

1 Like

Smug jab aside.

I have taken a look at the Digikey libraries. Great resource, of course. However, they also demonstrate my point regarding lack of standards working against semantic meaning and more.

At the ROM level (Rough Order of Magnitude) you have a situation that, due to the lack of guidance and lack of required fields, anyone who wants to contribute libraries will end-up rolling their own decision.

So, if twelve other part distributors and fifty component manufacturers decide to create libraries for KiCad every single one of them will have a very different field set. It would be a nightmare. And you could not even begin to create sensible tools such as BOM generators and useful in-context semantically-aware tools. In other words, it’s formula for disaster.

Take the Digikey zener library as an example. Here’s one of the two parts in it:

The value of this zener diode is the manufacturer part number, which is the symbol name. The “Description” field has the type “diode”, sub-type “zener”, voltage (5.1 V), power rating (500 MEGA Watts?) and footprint. All rolled into one text field. Not quite useless, but there is no way anyone I know would want to display that mess next to a zener on the schematic.

How about the part number? Or is it the value field? Sure, if you are super familiar with zeners you might choose to display this. In my world you want to show the zener voltage and current rating. In other words, what most would take to be the semantic meaning of “value” for a zener and the information that is actually useful in understanding a circuit at a basic level based on what you can see on the schematic and nothing else.

I’ll ignore all of the Digikey specific fields because this discussion is about a minimal set of fields that would improve the out of the box experience and also provide sensible guidance to enhance the ecosystem of tools and libraries out there. I think making these changes and publishing a guidance paper would truly open things up for companies like Digikey and other to contribute in unimaginable ways.

That said, the problem with the Digikey specific information is that, once again, it is based on some randomly chosen structure that makes sense to them. If every library was like this it would impossible to harmonize a set of tools to improve the KiCad experience at the basic to intermediate level of the user base.

I looked elsewhere. Fuses:

Again, semantic failure. Whoever wants to put “3413_0328_22” next to a fuse on the schematic, please raise your hand. Yup. That’s what I thought.

Throughout this discussion I have been consistent in my counseling that semantics (the meaning of things) matters, a lot. Without meaning you end-up interpreting things in randomly unimaginable ways. The lack of proper semantics and standards also means that you cannot deliver software that takes advantage of such standardization around the semantic description of the components we use. The easiest example of this being sensible treatment of a Bill of Materials in software.

If anything this discussion has caused me to further solidify my thinking in this regard. I have not found any counter argument that I would consider sound and valid. This is particularly true when you consider KiCad as the guiding element for an entire ecosystem that is developing around it as well as the value it does and will have in educating both hobbyists and future engineers. In this context, we have to push for base-level semantically meaningful representation of components in order to empower hobbyists, students, entrepreneurs and professional engineers to explode with creativity and make what will likely be incredible contributions to this ecosystem.

Professionals with the resources and know-how will always be able to do something else. And v7 seems to be on track to deliver tools and handles for them to do just that. This discussion isn’t about fixing it for professionals or forcing anyone into a workflow. It isn’t about having the KiCad team create libraries with millions of components. It is about fixing the very real problem about not having a minimal set of semantically relevant fields to describe a component in the out of the box experience and for new to intermediate users. That is lacking.

2 Likes

Yup, for all the reasons you stated. It isn’t a solution at all. It’s a place to make costly mistakes.

This, and the manufacturer field for disambiguation as well as unlinking the value field from the symbol name, would revolutionize what libraries and tools would be available in the ecosystem. Once a minimal standard is in place people and companies will have a compass to use for base-level contributions. Professionals will be able to take this and fly with it, enhancing it beyond the needs of the average user to fit their process and systems.

The concept is simple: Provide for a way to describe a part, at the most basic level, in a semantically meaningful way. Everything else grows from there.

And the user can stuff it with their internal part number if they wish. Or leave it empty.

If the part number field is also integrated into the API, more advanced users (or plugin writers) could use this value to trigger access and search into a database or ERP system of any kind. It could be as simple as a right click on a component to access a context menu that then makes a call to the database with the internal part number stored in the standard “part number” field.

The reverse flow is also possible. If the field is standard and it is used, a plug could interrogate the schematic or PCB from the context of the external database and do magical things.

I am not sure why people get worked up about the name of a field. Particularly something like “part number”. It doesn’t mean “Martin’s part number”. It just means “this is where you put the part number”. You could ignore it. You could stuff it with your own part number. Or you could populate it with the orderable manufacturer part number. Why so controversial?

Same with “manufacturer”. Ignore it. Stuff it with “self” or something to indicate the part number is internal if you wish. Or you could stuff it with the actual manufacturer name.

Having these two fields as standards can open the doors to all kinds of useful and interesting things. I would also fix the broken value field. It has to be fully independent from the symbol name.

yes and my point is, that there is no standardization possible or even needed, because everyone wants to name this field different. adding a standard field which won’t be used out of the box is useless. the only thing you change would be instead of clicking the plus button for the new field and name it like you want is changing it to: double clicking on the existing field to rename it, if it is used. if it is not used, one has to activly remove this field to prevent questions from other parts of the production line. It just generates more overhead to have this field from the beginnin then adding it if you need it yourself. I do this all the time. its not that complicated, I believe in you, that you also can manage to do it.

Whoever wants to put “3413_0328_22” next to a fuse on the schematic, please raise your hand. Yup. That’s what I thought

this quotation belongs to the “symbol name == symbol value” theme. If you read all answers slowly you will see (if I count correctly): the majority of the readers is with you that the historical connection between symbol name and symbol value is outdated. You have won this point already (most probably), no need to fight for this (again). Which priority this gets for development and how easy (and fast) this connection can be freed in the kicad-code is another question.

. I have not found any counter argument that I would consider sound and valid

I think my counter-arguments are good and I find them valid. They were written by me and many others many times:

  • the fields are not necessary for everyone (you said that multiple times, but it’s not true)
  • despite saying “they will not hurt anyone”:
    • they need screen-space in every symbol properties dialog which I would like to spend for other things. An extremly bad example for this was Altium Circuitstudio (yes, not the fully fledged AD) with >= 10 predefined, for me/my company wasted and annoying attribute fields)
    • more unnecessary fields - more chance to write my information into false fields. Yes, that would be my fault.
    • they are exported with the bom - it’s my time to delete these on every bom-export. I don’t want to spend my time for your desires
    • despite your saying “these fields are semantically defined for xyz” they are not immune against abusing
  • each new fields which define a “standard” are only one standard more (you mentioned you have programmed something. You know the xkcd-comic regarding standards?)
  • even if you define two new fields: it’s not enough - compare with the digikey library. They needed 9 fields. So your wanted “standardization” fails because all third-party-libraries still use different library-fields.
  • the “out-of-the-box”-experience:
    • a beginner has 1000 other things to consider/learn than MPN-numbers
    • a student: To add fields themselves gives learning about the functionality - not only “filling excel-sheet-fields”, but learning how things are connected together (Ahhh - two fields → nametag<->vale–> student has learned)
    • companys: out-of-the-box experience is not so relevant. The employer has to use the tool at hand.
  • my biggest counterargument: it’s already easy possible for everyone to add fields as they wish
  • and especially for companies with >=20 hardware-developers there is possibly also one person who is able to write phyton-scripts and adapt all libraries to the company-guidelines. running these script automatic after each install - that should be not to hard

edit to add a last sentence: Like craftyjon said (1 answer below) I think I also understand your point of view. There are always advantages of standardization. It’s only that I think the advantages in this case are very minor and for my point of view the disadvantage outweights the advantage.

1 Like

I just want to make clear that I completely understand @robomartin 's proposal and desire for a more rigid semantic system of properties/fields. I don’t think this will be implemented in the short term, and maybe never, but I do understand it.

If we did implement something related to this, it is more likely that we would implement a system allowing someone to define and enforce their own semantics, rather than building a single rigid set of semantics into KiCad itself.

Some of the other points on this thread are known issues that we do plan to improve and/or are already working on, such as:

  • It should be easier to configure BOM exports without needing to write Python code (#9992, #5408, etc)
  • It should be easier to manage library metadata (#11506, #4127, #2141 and many more)
  • It should be easier to implement fully-atomic libraries (#7436, #6941, etc)

I also want to note for those who may not be aware that the KiCad lead developer team does not control the KiCad library project – these are all part of the KiCad project, but a proposal to modify the KiCad standard library content or practices (the KiCad Library Conventions, or KLC) is a question for the library team, not the KiCad developers.

I too am familiar with military / aerospace design requirement. I can’t imagine using Kicad in a integrated system meeting such requirements. I’m thinking you are are talking about some combination PLM and Mfg system.

“on the fly” Back when I was the engineer designing circuits I would start with a schematic (then on 11 x 17 paper). As I roughed out the design I would add resistors and capacitors etc. The detail of these components were not of concern at that time as I was only interested in the basic function. When I thought the design would actually function I would go back and look more closely at the values and components.

As opposed to having a fully functioning tested system and going to a design house to have a PCB made.

BTW my whole career has been in new product design. Most of the challenge has been the unique capabilities of our products. This requires understand the system our product was part of but I’ve thankfully not been that deeply involved is a design phase on your referenced page 20 diagram.

The excerpt you quoted is an example of how overloading a field, like the value field, is a bad idea.

I’m not sure I understand your statement “overloading a field”. Or what it is you are requesting. The value field is just that. It could be ohms, farads or even the base number of an IC. For instance I might use the value field with a base number of an IC. say TLC4040 for a battery charger IC. While this is not a numeric value is it however the general descriptive name of an IC. And it keeps the full part number off the schematic graphic. And if I want to use the value field for some personally cryptic information then that’s my prerogative.

I am all for standardization, having well defined interfaces, and a solid database / library system that can be used by both manufacturers, big shops and individual companies but that has nothing to do with adding default empty fields to all the library symbols.

To me it makes more sense that a company that already has a database (for some other EDA suite) with some set of semantics to keep on using the same semantics for KiCad, instead of juggling between their database semantics and a different set of names for KiCad.
Those company databases are a very valuable asset for them and KiCad should have the flexibility to conform to that instead of mandating some other set of names.

Having a list of preferred semantics for use in KiCad is a good idea to have some guidance for people who want it, but hard coding them into KiCad’s libraries is not a good idea.
It’s also quite easy to write a script to replace one set of keywords to another set in a library, and having a mechanism in place for this is much more practical then hoping the whole world will stick to the same convention. We can’t even agree to use the Metric system. (That’s not really true, almost the whole world uses Metric, except a single significant country and some small area’s).

1 Like

Yes, of course, that’s a normal process. I do that all the time. Most of my designs start as pen-and-paper ideas. This applies to software, electronics and mechanical. I still do state machine and such things on paper before coding. It is often hard to create in the context of a rigid CAD system of any kind, I see it more as a means for documentation once you have a reasonable idea of what it is you are after.

What I will sometimes do on EDA is grab components from the most convenient library and thread together a design. That means I give preference to our vetted components, sometimes without regard for values or details and, if they are not there, I might go for a stock library of some kind just to get moving. The closer I get to the golden component library the less work I create for myself later.

At some point all components have to be from the qualified libraries. If some are missing, this is the moment to do the work require to add them.

Once you do that and make all appropriate replacements it’s one turn of the crank to get a BOM you can review and order and off to layout it is.

I would guess everyone has a process along these lines to one degree or another. You have to get your thoughts down first before you can engage in detail work.

Ah, good point. I am using language from object-oriented programming without realizing a lot of people might have no clue what I am talking about. Thanks for pointing this out.

In case you happen to be interested in that context, Wikipedia does a decent job:

https://en.wikipedia.org/wiki/Function_overloading

In this context I use it to mean the use of a field for more than one thing and that these things are not necessarily the same.

Let me use a completely unrelated example:

You go to the doctor. They make you fill out one of those horrid forms nobody likes to fill out. One of the items has a label that reads “height”. You write in your height and move on.

A few minutes later the assistant comes over and says, “You didn’t enter your weight. We need that.”.

You look at the form. There is no field tagged “weight”.

You ask. They say "Oh, just put it in the “height” field.

What do I do with my height?

Oh, just cross it out.

Do you need it?

Yeah, but there’s no place to put it. Just write it somewhere on the side of the page.

So…you want me to put my weight where it says “height” and my height on the side of the page?

Yes.

OK.

The reuse of the “height” field to record the weight is, in programming terms, akin to function overloading. It isn’t a perfect parallel but most programmers would understand what I meant when I used the term. Sorry for not realizing I was speaking gibberish to everyone else.

Anyone can understand the obvious problems with such a medical form. Imagine if every office used the “height” field differently. Imagine if every office had a different idea on where to record the weight and what to call it. It would be a mess and your medical records could not be moved shared from doctor to doctor without a ton of work.

That’s the issue I am presenting here. The term “semantics”, as I apply it here, is about having fields that capture the appropriate meaning. The “value” field should never have a part number or be linked to the symbol name. Anyone can understand that the “footprint” field should not have the URL to the datasheet and that the “datasheet” field should not have a link to the 3D model. Everyone gets this and would likely agree with this. Well, it isn’t any different with the “value” field.

From this it follows that we need two standard fields: “part number” and “manufacturer”, along with a “value” field that has no connection to the symbol name.

This isn’t about forcing people into workflows as much as it is about creating a sensible starting point from which a greater ecosystem can evolve. Nobody in the electronics industry orders part by hand-waving. As a famous race car builder and mechanic used to say “When all the smoke and bullshit clears out, you have to drive the car and win the race”. Well, when all the huffing and puffing clears out, you have to have a real orderable part number and the manufacturer’s name to be able to get anywhere with anything other than hobby electronics or just hacking around. They are, without a doubt, absolutely necessary. And KiCad does not provide official support for any form of this in a standardize manner that would enable the user and software author community to expand capabilities beyond what anyone could possibly imagine.

I think I made this clear but I’ll try to clarify further here in case I haven’t.

I am not proposing anything rigid be put into place. Flexibility is key or you won’t be able to address the myriad needs out there.

What I am suggesting is that a starting point that addresses the deficiencies I have highlighted would be important in order to greatly enhance usability and enable an ecosystem of libraries and tools to grow around this baseline.

Advanced users should be able to completely ignore this and do as they wish. No question there at all. I think it is precisely the newbies, students and intermediate users who would greatly benefit from addressing these issues in a sensible and flexible manner. As students become professionals they might very well adopt different methodologies and adapt to whatever it is their company likes to do. Nothing in what I am proposing challenges or negates this level of flexibility.

The kinds of changes I am discussing require matching functionality at the application layer. I would imagine the library team has nothing to do with this. Right? And so, if the library team decides to add fields that require intelligence within the application, unless this work happens, the addition might not have the desired effects.

As you have dscovered it only adds the field for the symbols where you fill in the field. If you were populating the schematic from an enhanced DB where parts have IDs then the field would be filled in.

Users should not push the contents of the part field back to the library, that’s the job of the librarian.

1 Like

Or you could use a configurable BOM generator like KiBOM which can adapt to the actual names of the fields and output the CSV with different heading names. You’ll need a flexible BOM tool to handle the different column names and orders required by various assembly houses anyway. It’s also likely that such a BOM tool will have to do more lookups on the DB, to get the MPN for example. So you’ll have to base your argument for a standard name on other reasons.

I agree there should be a part ID field. I doubt if the developers will want to standardise on the name. In the absence of that, you can just publish your software and by being the first establish your chosen name by popularity. It doesn’t have to be a progress blocker, just pick a reasonable name and if the developers agree to a standard name later, just do a global edit.

I agree that there should be standardized field names for Part_Source or MFR and Part_Number or PN or something similar. They should be left blank in the standard libraries, but having standard names allows the software to do things to support the use of a part number, be it a manufacturer’s part number or a house part number.

There are issues with the proposed alternate of just adding these fields to your own custom libraries. For example, I copied the L272 dual op-amp from the standard library into a personal global library and edited the symbol to add the two new fields. Then I placed all three sub units on schematic and assigned them to U9. I set the part source and part number fields on the first unit, U9A, but those values were not displayed when editing the second unit. I could then add different part source and part number values to these fields for the other units, U9B and U9C. This leads to the three sub-units of a single dual op-amp part showing different values for the part source and part number as shown in the screen shot below. (Note, the TI and onsemi part numbers are fictitious).

This is clearly not right. The software should know that the three units in a part can only have one part number. It could display the part number values for only the first unit or for all units, and allow editing from the first unit only or from any unit. If it does the later, it should update the one and only part number for the part and update the displayed part numbers for all units. Editing the part number for unit B should change the value displayed for unit A and C.

The current situation arises because the software does not attach any semantic meaning to the part number added by the user. It is simply a string that can be displayed beside the part’s symbol. Right now it doesn’t know that all the units in a part must come from the same part with the same part number. This could be enforced by KiCad if the part number were always there (even if blank) in a field with a known name.

I’m sure that other uses for part number fields would become apparent if they were always available.

I think the KLC could be amended to add recommendations for library maintainers to use these fields correctly to build consistency between third party libraries.

It’s a simple change to the KiCad standard libraries which would encourage uniform use without undue burden on the library maintainers. The changes to the KiCad software could be made as time allows since they are not needed for the new fields to be used, but might add new features or checks in the future. No one would be forced to use the new fields.

I don’t think this proves that anything is wrong with user defined fields because you started from the library end. If you start from the schematic editor end the results are different.

I took one of my designs and added the field LCSC to it. Next I went into Tools > Edit Symbol Fields and added a random part number C244. When I edit the properties of the (unit of) the IC in question, there is only one LCSC field no matter which unit you choose. (Similar to the datasheet field.)

A correct implementation would populate the part number from a database, not the library. So another type of library would be needed. There are already KiCad libraries and KiCad legacy libraries. It might also be necessary to tweak the application to know about read-only fields.

If anything this shows that the stock libraries should not be modified. If anything the additional fields should come from a full-fledged database.