Semantic problem with KiCad component treatment?

Software
86

I know you now understand what I am proposing. Let me comment on this one.

Yes, this is convenient and useful. This is where, again, to overuse the word, semantic coding is useful and important.

You could have all kinds of fields that, along with their semantic meaning, could trigger UI elements and other functionality in the program.

For example, if you had part number and manufacturer fields --and they were filled out-- you could show a “Search” or “Price” menu just as is the case for the datasheet field. You could further provide a place in settings to setup your favorite meaning for these contextual functions. You might decide you always want to search Mouser for your “Price” but would rather search the web for the combination of part number and manufacturer.

One use case is that you often have to go to the manufacturer for simulation models, special footprint details, evaluation board or circuit examples, application notes, etc. I do that all the time. The datasheet isn’t always a complete resource. The only way you get there quickly is with a part number and a manufacturer and, like I said, intelligence within KiCad to allow you to do something with them.

This is why I told @retiredfeline that just writing a bunch of code in Python isn’t a solution. There’s a lot more to it than meets the eye.

87

You never answered my question countless posts back about why additional fields aren’t sufficient and a first-class field would be. Nonetheless I think you would actually get somewhere by programming instead of writing reams. You might be able to use the experiences to justify a feature request.

Edit: So you want more support for the fields in the UI. This can be accomodated in a more general UI, and can be a separate request. It still pays to keep the framework generic. You may think that once your requirements are implemented, that’s the end of the matter. Unfortunately any specialisation of function will add baggage against a more general solution in future. Sometimes more becomes less in future.

The embedded Python scripting can also be used to implement some of the things you suggest. That’s why one powerful general feature is worth more than lots of small special features.

88

Nope. That’s not true. That is not what this feature request would be about. Put simply, this is what it would cover:

  • Add a part number field to every symbol in the stock libraries
  • Add a manufacturer field to every symbol in the stock libraries
  • Both of them blank by default
  • Add semantic functionality to support these two fields if used (I gave an example of this in another reply)

That’s basically it. I need to learn more about the link between the value field and the symbol name. I haven’t had the time to play with it but, if they are linked, this is a bad idea. Value --again, semantically-- has nothing to do with the symbol name. If they are connected all it is doing is confusing matters. I understand that it can be changed at the schematic level. If it cannot be different from the part name at the library level this needs to change and I would add it to the feature request or create a separate one (might even be bold enough to call it a bug).

And you are wrong. Just the addition of these two blank fields would be massively useful for all kinds of users, even professionals. The existence of these fields would allow for sensible treatment of BOMs and other very useful tools, some we might not even be able to imagine right now.

It’s a simple change with a huge positive outcome in terms of the out of the box experience, usability and standardization.

Anyone who doesn’t like it can do whatever they want. They can do that now. No difference to them. Huge positive difference for all others.

1 Like
89

You’ll get even more pushback against this. Volunteer resources are stretched enough as it is. And it’s unnecessary to modify the stock libraries. You will be using your own database (when that’s implemented). So it will be your responsibility to import the stock libraries and populate the extra fields however you wish afterwards.

Edit: Even now, it’s not necessary to modify the library to have extra fields. The extra fields are specified in the per-user configuration, the eeschema.json file I programmed a script to modify a while back. IIRC there’s a sample Field4 in the stock configuration and I have added other fields for the purpose of generating BOMs while still using the stock libraries. So maybe you need to observe how the system works first.

1 Like
90

Things like prices also do not have much use to add to the KiCad symbols themselves. The big stores do not like it if automated tools crawl their prices and they actively fight against it by constantly changing names and other small things that are invisible to users but break crawling tools. This has been tried with KiCost.

Prices also fluctuate too much to be very useful to add them to the symbols themselves.

I do see an advantage of a list with semantic names that have a special meaning, but that does not mean they have to be added by default to all KiCad schematic symbols. For example, I really see no reason for the “Datasheet” field to exist at all if it is empty.
For example a drop down list to add those fields if needed is a flexible system and it does not need to add empty fields to all the KiCad libraries. There are currently some 17000 schematic symbols in Kicad’s libraries, and adding a bunch of fields to all of them is nothing but a waste of space and makes KiCad slower.

Having an implementation that can work with a list of such names and semi automatically adds them to schematic symbols when needed is much more flexible then a static list added to those symbols, it avoids the clutter in the schematic symbol libraries themselves, It’s easier to extend and it can even be implemented in a way that is transparent to the users.

91

I’ve never figured out what the exact link is between symbol name and the Value field.
In older versions it was mandatory that they were the same on a library level, but could be changed in the schematic. I think this has already changed, but I’m not sure about that.
I agree that a hard link between these is a bug that should be fixed.

Nope. They do not have to exist at all.

For example:
Schematic Editor / Tools / Edit Symbol Fields
Press the Add Field button, type a name and press [Enter]
You now have an extra column, and if you add data in that column, then the new field you created is added automatically to each schematic symbol for which you entered data.

Improving the tools in this way is a lot better then adding static fields to all the library symbols.

92

Sorry, I guess I missed it.

Simple example:

If the part number and manufacturer fields are filled you could:

  • Right-click on a component and trigger a web search of the manufacturer’s website
    • You have to go to the manufacturer website for:
      • Simulation models
      • 3D models
      • Application notes
      • Samples
      • Technical drawings (example: a connector has a mechanical drawing and a datasheet and more)
      • Development boards
      • Equivalent components/substitution
      • Support
      • In the case of MCU’s, FPGA’s etc. development tools and support
      • Etc.
  • Right click on a component and perform a pricing search
    • You could have this be configurable in settings or through a JSON file
    • You could have it search your favorite vendor or something like Octopart
    • You could also have it launch an external script to do interesting things
      • For example, price the part at Mouser and Digikey and pull in the 1000 piece price into a custom field
  • Right click on the component and trigger an external script that looks it up in your database

Just three quick thoughts. One could do a lot more. However, these fields need to be semantically integrated into KiCad so that, if you use them --remember, I am saying they are blank by default-- the software and the user (through external applications) can make intelligent use of them.

At the most basic level the fields can be completely ignored and nothing whatsoever would change for any KiCad user today. No difference. Just two fields you never use. I seriously doubt most people will ignore them, BTW.

Next level is, you use them to store the most important information you need in order to actually build a board. And, as a side effect, you get to use a few new features that are enabled when the fields are populated.

The third level is a connection to external scripts that you either register to show-up on right click or can reach into the design via an API.

I am pretty sure all of this requires work at the core level, rather than writing a bunch of external Python code. I have not done enough work writing Python for KiCad to know this conclusively. I suspect I am right on a good portion of that assumption. In fact, I am about to launch into a more serious effort with Python/KiCad and need to post for help setting-up a solid development environment under Windows with Pycharm or VSCode (subject for a different discussion).

EDIT:
I’ll add one more to the list. If you have these two fields and they are accessible via the Python API, you could easily integrate these into a database driven process after populating the part number with your internal part number (or just use manufacturer and their part number) to pull in relevant data, instantly create and populate new fields in the schematic and do just about whatever the API allows in both schematic and PCB.

The key is to have these two fields as standard and fully-integrated fields. They can be blank and completely ignored by most. And yet, if sensibly integrated, they functionality they would enable and the experience they would provide would be a vast improvement over the current state of affairs.

Instant databased-driven functionality. Just two fields.

93

I have done this. This is not a solution. Read my other posts. And this does not improve the out of the box experience at all. For example, one can’t write standard add-ins to handle BOM creation using these fields. I had to go and manually create my own code to extract these into a BOM. That isn’t what a good out of the box experience should look like. And, of course, there’s a lot more to it than just the BOM.

1 Like
94

I read back the first 20 or so posts and you are mixing up a few things.

The current KiCad libraries are not a fully specified database, and they probably will not become so in the near future. Adding some fields to them is not going to help.

The database, and the KiCad libraries are not the same thing.
a short piece from one of the first posts:

The current KiCad libraries are simply not fit to be used as the database you want, and adding some fields to schematic symbols is not going to change that.

It makes no sense to add empty fields to the KiCad schematic symbol libraries, and duplicating a resistor 100.000 times for the different manufacturers, types, values also does not make sense. The KiCad project simply does not have the resources to maintain such a database (yet…).
The only thing that makes sense to me is to have a separate database with metadata such as part numbers and have a mechanism to combine the “database” with the “libraries”.

Maybe you started too early with KiCad and should have waited for V7. Maybe you can help in improving the implementation for what this is going to look like in KiCad V7.

Such a database will also not become a part of KiCad, simply because nobody in their right mind would trust it, and would have to verify it for themselves anyway. Your company has spent countless hours creating, vetting and verifying their libraries.

The Semantics is a different thing. In the basic form it is only a list of agreed upon names with specific meanings. It is quite simple to write a script to match an existing and trusted database and combine that info with the (mostly graphical data) in the KiCad Libraries.

The KiCad libraries are not the database you wish for, and they will not become that database.
The KiCad libraries are only a building block of a greater part.

I’m guessing that the database driven part in KiCad V7 is going to have a reference implementation of combining an already existing (and vetted) database of “golden parts” with the graphics in KiCad’s current schematic symbol and footprint libraries, combined with tools to help with further verification and selecting. This way you only have to verify a very small part of the KiCad libraries, and then combine it with the already verified and trusted database.

1 Like
95

I’m a little late to the party but…

I guess I don’t understand your goal. IMHO the schematic is not the place to control a production device BOM. And certainly not provide inventory quantities.

If I understand you correctly, you would need library entries for every part you could use, every resistor value and capacitor value. I don’t see how this is practical ( or possible).

What is your use case? Are you handed a fully detailed design with part numbers etc? Or are you designing “on the fly” (this is what I do).

Just MHO

96

Sorry, you are mischaracterizing what I’ve been saying and continuing along a line of discussion that is pointless for me to pursue. Unless you have something relevant and useful to say I am going to look past your comments from here forward. Other than that, thanks for participating. It has been truly enlightening.

97

The excerpt you quoted is an example of how overloading a field, like the value field, is a bad idea.

No, I have no interest in keeping inventory data in the schematic. That would be silly. The place for that is in a suitably designed and linked database or, at a higher level, ERP system.

I don’t think I have ever been handed a detailed design in 40 years. No, I typically design devices from scratch. That process typically starts with developing specifications, listing desired functionality, reviewing trades, suggesting devices or parts of interest, etc. An iterative approach follows this and eventually lands on electrical, mechanical, software and other specifications. From there you break out into the different disciplines and execute as needed, each one following a similar iterative pattern.

Depending on the type of project, a reduced or expanded version of the process you’ll find depicted on page 20 of this manual. For example, commercial projects like the one I am working on now don’t require this level of detail and rigidity. When I was working on a camera system to send to the moon (fingers crossed, in the next few years) it was pretty much at this level.

https://www.nasa.gov/sites/default/files/atoms/files/nasa_systems_engineering_handbook.pdf

Could you clarify what you mean by designing “on the fly”?.

98

Sounds like you are not a programmer and don’t understand that Python is embedded in the KiCad application and has access to all the data structures, and is not “external” in the RAM domain sense. More research on your part is called for.

99

I don’t know. I’ve only been writing software for some 40 years? Machine language, Forth, C, C++, LISP, APL, Objective-C, PHP, Javascript, Python even Visual Basic! Embedded, desktop, web even a series of educational apps for kids on the iOS platform way back when. Oh, yes, and FPGA design in Verilog, which isn’t a programming language but, well, there. Hell, I ever published a paper on a programming system I developed for robotics back in the 80’s.

Yeah, you are right. I have no clue.

100

It’s the same idea as the JavaScript embedded in your web browser so you actually have come across the concept. BTW there is no Python API for the schematic editor yet, that’s another thing that was pushed back to 7.

So I reiterate, you can already do what you ask for in 6 and the rest in 7, you don’t need any special considerations.

You can add a field to symbols in the schematic editor, no need to touch the stock libraries. In fact having a couple of placeholder blank fields is a waste of effort since to be able to populate them you need to have write access to the library, so a writable copy has to be made. Then you have the problem of ensuring atomic updates, so eventually a DB is called for.

Python as a scripting language with the appropriate API hooks into the editor is a more flexible and sustainable way of doing the fancy things you suggest.

Also, Tim B-L isn’t god. The idea behind the semantic web are valid but a lot can be done by other means. (So much so that nowadays the concern is that The Man knows too much about you.) You do not have to have known field names. The field content can encode both more than just an identifier. You could attach a mime type, you could even have a snippet of JSON in it. What does it matter if the content isn’t a simple identifier in the database? It can be opaque and the Python scripts can decipher it.

Then too, having a part and manufacturer field is not good DB practice. You have redundancy and possible inconsistency. The part should be used as a key to look up the rest of the fields; that’s what joins are for. Looked at this way, the datasheet field becomes redundant.

So I say, a mininum of special meaning fields, let the user add more, and let all fields participate in “first-class citizen” actions whatever that means. But I would not go as far as just one UUID field.

101

@robomartin I know it isn’t always easy, but if you would’ve explained in your original post your vision and the benefit with your suggestion I strongly believe this thread had taken a completely different direction. I’ve read through the entire thread (although not carefully so I’ve probably missed a few things), and it took me until post 88 and post 92 to understand your suggestion.

I think you had us fooled by mentioning “out of the box experience” a number of times. That led me to believe that you suggested adding data to the standard libraries.

I can understand the benefit of adding the two fields if:

  • Left blank in the standard libraries.
  • At least some basic functionality relies upon these two fields.

I wouldn’t use them myself since I have fully atomic libraries with House Part Number in the Name field (and Value field since they are the same in the libraries). But I wound’t be opposed to adding this functionality into KiCad. I can see the benefit and I could easily ignore these two fields.

In every single symbol in our atomic libraries I’ve filled in the Datasheet field with a link to the component in our web-based product management system. That way it’s really easy for me to just select a symbol in the schematic and press “D” to get all it’s data directly in our system. Some of the things you’re suggesting is sort of similar to this.

You are right that the BOM export isn’t optimal and there is work planned on this. I’m not sure, but I believe that it’s planned for V7. Personally I love it like it is. I’ve written a Python script that gives me exactly the result our Product Management System wants and I never have to think about what’s being outputted. But I totally agree that the “out of the box experience” with the BOM export isn’t great.

But this is another topic. Improving the BOM export has nothing to do with the suggestion of adding the two fields. Improving this should be done anyway. I might add other fields to the symbols which I would also like the BOM to include, and IMHO the BOM export should have an easy way to include these.

1 Like
102

I also disagree with the fact we need standardization here, mostly based on two facts:

  • everyone wants to name the part number field differently. some use abbreviations, some special naming etc. all of these would have to adjust this new mandatory field to their liking and would have to ignore an error thrown by the KLC checkers when handling their symbols.
  • not every part needs a part number. there are many parts which are not connected to components at all (testpoints, graphical elements, special layout components e.g. for RF), or components where you intentionally don’t want part numbers, either because they are only placeholders for later versions/other populating options or because they are wildcard components where it does not really matter which kind of 10k@1% resistor is placed there. especially the last point is getting more and more important with the shortages we see in all different components where you simply have to adapt your components on the fly to the market situation. for these cases you would get another empty field additionally to the datasheet one which can lead to different time consuming questions down the line of your production process.
103

I agree that having some standardized names makes it easier to improve BOM tools and other parts, and I think that is all you are trying to say. (In this ridiculously long thread).

Adding the empty names directly to the KiCad libraries is useless.
You’ve never given any response to the Idea of defining those semantic names as some sort of “automatic” variables and tools to work with them.

The database driven ideology is also never going to work without proper integration. If schematic symbols have part numbers but you can still change the value independent of the part number then stuff gets out of sink and data becomes unreliable.

At the moment I’m not even sure if we’re thinking along completely different wavelengths, or if you are deliberately ignoring most of the responses or even trolling. I do think you are using far to many words to describe simple things and apparently still are not able to communicate clearly. And from the other posts I read I’m not the only one thinking this.

1 Like
104

I don’t think you’re wrong about adding these fields as defaults. Where you went wrong is thinking the developers care about what’s written here, or anything really that isn’t a well-written code contribution. All this typing is just spitting into the wind.

105

I only agree with you here, your previous sentence seems invalid to me. Beware wind changes direction sometimes.

Who here wants to create a library as the OP describes and maintain over time? Feel free to do so and share with Package and Content Manager.

Why do you think it is ok to have empty fields? If someone comes to the conclution that those fields are mandatory, then those can be created to cover the need… Could that be more simple? This is not KiCad’s inability to cope with extra fields, to me this looks kind of, someone else should do the dirty work for me.

OP since you know your way around software dev, help make this great program even better, a thread like this about two empty strings, could be also considered funny… :wink: