Libraries and Components- The Disconnect


#41

Companies I’ve worked for use “trade” distributors like Arrow, Avnet and a bunch of others most people have never heard of. Larger companies go further up the distribution chain up to the manufacturer themselves.

The design engineers select parts and specify alternatives, but it is up to the Purchasing department which variant they buy, and who from. They do not use the CAD tool that the designers use. In fact there is a completely separate software package that Operations use for ERP that integrates purchasing, stock control, production, sales orders etc such as SAP.

Component data is kept in a separate database, which has additional info relevant to designers, but is linked by “house” part number to the main ERP database, and kept in sync. Data from that database is then extracted to create libraries for our CAD tools (schematic, layout, MCAD). The CAD tools used by the engineers are just a cog in a big machine.

Obviously we have an IT department and other “programmer/engineers” who maintain the databases and bespoke scripts.

All the above can and is implemented outside the CAD tool. Therefore in principle, KiCad already has the features needed. All it needs on the user side is some method of generating libraries for KiCad, which is dependent on the local requirements which really only the user can implement. It’s not practical for KiCad to contain interfaces to every type of ERP tool and every electronics distributor.

I think what people are really looking for is a free ERP tool, that can be adapted for use with KiCad.


#42

Yes, where possible we deal directly with the manufacturer or one of their reps, otherwise we use a distributor. It usually doesn’t do us much good to scrape websites as that is seldom the price we pay anyway.

Quite the contrary, I was only suggesting being able to use a plugin with ESchema to allow the user to implement, or interface to, the database or ERP system of their choice. As you say, much of this can already be done with external tools, and generating libraries is certainly one way. My suggestion was simply to facilitate a minimal integration between the CAD and ERP systems. This requires nothing more from KiCad than Python scripting support, which I thought was planned for V5, if not then this discussion is rather moot anyway.


#43

Perhaps I should have said “some people” :slight_smile:

The thing about an integration API is that it would affect a lot of KiCad code at a core level, ie component chooser,netlist gen,symbol editor,cvpcb,edit footprint assignment,read netlist,place module. Currently KiCad reads all libs into a memory cache, that would not scale well to very large databases. There need to be some fairly major internal architectural changes before a clean API could be implemented.

KiCad does have a “plugin” architecture for reading symbol and footprints libs, but these are internal to KiCad and not accessible by users. These plugins are called on a one-off basis when the internal cache is built. However, it should be possible to create a generic plugin that reads from a database instead of a text file. The model here is that the github plugin actually downloads a zip via the web.

The scripting support in v5 is limited to pcbnew, basically a script can access all elements on a board, but that doesn’t give access to symbol or footprint libraries (AFAIK, I might be wrong). Of course a script either run inside or outside KiCad can access external text files, but there lacks a method for a script (or the user) to tell KiCad “the external files have changed, please re-read them”.

For all these reasons, I try to gently steer people away from the expectation that KiCad should (or will) have a built in database functions or direct links to supplier websites. Obviously I don’t speak for KiCad devs in any way, or even always agree with what they decide, but I have been watching them for a few years and have a little knowledge of the codebase. I can generally predict what they will say regarding new features.


#44

I came across this project recently https://github.com/turdusmerula/kipartman/issues/18 - kipartman - which I guess has some of the bones of what several contributors have in mind. I have not been able to get it to run and there are some limitations apparent but I think the concept is quite interesting.
With a python API to eeschema this would all be a great deal easier and would facilitate a variety of options for linking in to external part databases.


#45

You are probably correct about some people wanting an ERP system for use with KiCad. I didn’t know if your reply was to me or not but in any case I did not take it personally, I just tried to make it clearer that I was only talking about interfacing with existing database/ERP systems.

I was thinking about a little less integration than that. My thoughts were that KiCad would continue to load libraries as it does, only the component chooser would be populated with parts from a database. Once you selected that part the plugin would use data from the database to load the appropriate symbol from the library and assign the appropriate footprint to that symbol, as well as any other fields the plugin read from the database (ie. value, tolerance, wattage, voltage …). Instead of selecting the default component chooser you could select the plugin which could present it’s own chooser dialog.

Anyway, those were just my thoughts, none of which is going to be possible until ESchema supports Python plugins. However, after giving it a bit more thought at the gym tonight, and borrowing from what you said earlier, perhaps a more doable approach would be to create an external tool that would read a database and build the symbol library accordingly.


#46

I think I agree that Python support in eeschema would be a basic first step, and maybe sufficient. The pcbnew scripting has been really useful and I think the ActionPlugin will be another good step forward.

I know Python for eeschema has been talked about informally, but it doesn’t appear to be on KiCad roadmap (either v5 or v6). There have been some grumbles about the SWIG approach, it may not be perfect but I find it gets the job done.


#47

There is a programming concept called the “Unix Philosophy” that says to keep all programs simple. Each program should do one thing and do it well. You build complex solutions by linking together simple tools. The other way is to make your tool do all things for all users. You add features that are far from the core objective and it becomes bloated and hard to manage. Kicad is heading down this path. You should think about adding new features by adding a new tool rather than shoehorning into an existing one.

Eeschema should be stripped down into eeschemaLite with design management and hierarchy done in a separate new tool.

Selecting a part is a multistep process. One designer places a Nand gate. Another on packages 4 of them into a component. Another selects logic family,speed and temp range. Still another picks thru hole or surface mount. You want to support uplifting whats on the schematic rather than having to delete and replace with a new component.

John Eaton


#48

Yes, that is the problem.


#49

What I’d like to ultimately see is a way for KiCAD to access a database (SQL preferred) that could have many many components in it that ties vendor and distributor information to local schematic parts, symbols, and in-house part numbers. Then when parts are placed into a design 1.) all the vendor and distributor information is copied locally and is static, or 2.) some information is placed into the design and then BOM generation and other tools pull the information dynamically from the database (preferred).

Right now, if you wanted 1-to-1 mapping of placed parts in a design to order-able parts, your schematic library begins to balloon. Let’s take resistors for example. Right now on Digi-Key there are 860,049 different order-able parts. With a schematic library of maybe 20 parts and a maybe 200 footprints you could get near 100% coverage (while the database handles all the linkages).


#50

I like the process, I am just wondering in this scenario who creates the database?


#51

And maintains it.
I have been updating a BOM from 8 years ago that referenced RS and Element14 part numbers.
Nearly all of them are still stocked, but the reference has changed. Fortunately intern hourly cost is low.


#52

Database creation and maintenance would be left to users, much as library and land pattern creation is. Many medium and large companies have dedicated maintainers.

I think you would ultimately see people in the community step up and and maintain and create the database. Much of the work can be done via scripting and DigiKey scraping.


#53

I find that highly unlikely, since the official libs contain barely 5400 symbols and it is a struggle to maintain those, even with some scripting and community support.

If users create their own databases, there are dozens they could choose from, so KiCad might need dozens of interfaces. The database schema would need to be compatible with KiCad also.

I still think the practical way forward is for all part maintenance to be done outside of KiCad, and a custom set of libraries is created by each user for their own needs. Tools like KiField are already part way along that path.


#54

Big companies with their own house numbering systems have substantial component standards departments to maintain their databases. The coverage of these house number databases is at least an order of magnitude lower than RS or Digikey, as they only recognize parts already used in their own products. Expecting that volunteers will be able to do more than that is wishful thinking


#55

Crowdsourcing is a great idea; in practice it most often means that crowds try to find a source to satisfy them.


#56

This would do nicely for me. I can use plug-ins for everything except schematic input at the moment, so an interface that calls a plug-in and then parses the standard output from that plug-in as the part definition then I’m sure I could make it work. A bit of python to query a database and print a bit of formatted output would give me what I need. Then I could tie it in with the BOM plug-in and be able to build a good part database management system.


#57

I tried it :slight_smile: A quick script to parse the database and create a KiCad lib. It has severe limitations though, not least that the value field has to be unique as it pulls in from the library. So you can’t have a 1K 0402, 0603, 0805 and 1206 resistor all just having a 1K value. The current mechanism pulls in a generic resistor and then you edit the value afterwards. There needs to be a separate unique part identified that is not the value field.


#58

The hooks are missing, I can’t do what I want with what’s available.

The other downside is that there are various different incompatible approaches to a problem out there, just look at all the variants that people have thrown together to do BOM management. This is not necessarily a bad thing, but it means you have to find the one you want and stick to it.