KiCad nightly (v5.99): New schematic and symbol library file formats are now the default

Rene_Poschl · May 9, 2020, 6:58am

What would that mean? KiCad already supports fully specified symbols which kind of already do what most people expect an atomic thing to do but allow the library to be scalable (as the same footprint can serve multiple symbols).

Or do you mean something like a higher abstraction layer. A data container that takes a generic symbol adds BOM info and connects it to a generic footprint and possibly a spice model (I seem to remember @Seth_h suggested something like this, whereas i kind of assumed the inheritance feature would allow for this without needing a separate data container). If so please don’t call this “atomic”. Atomic means one file has all the info in it (it is inseparable).

Edit: A full atomic file format might actually make sense. But not for the local library or even a company library but as an exchange format used to get library assets provided by a manufacturer into KiCad (would be nice if a user would download one file and get symbols, footprints, spice models and 3d models installed by use of only one tool).

Footprints can have a differing value field from the footprint name why should this be different for symbols?

Rene_Poschl · May 10, 2020, 5:22am

A post was split to a new topic: Update PCB from Schematic warns that zones are not connected in nightly

uschmelmer · May 10, 2020, 6:37pm

@stambaughw its exactly what @rrascher mentioned. Having a unique symbol Like R_1K50_1%_0603_100PPM but with value 1k5 for readibility in schematic and layout.
What are the the “atomic” plans?
In Mentor Xpedition Enterprise we have the possibility of a unique “Part” (with unique inhouse partnumber and all values) and we map a generic symbol an layout footprint to it. that would be “THE” feature for kicad as professional tool.

paulvdh · May 10, 2020, 7:18pm

Both uschmelmer and rrascher are writing about an integrated database system, and I’ve seen several discussions about that both here on the KiCad forum, and on the KiCad section on Eevblog. Apparently it is a topic that is very deer to a lot of people, and I’ve several times seen strong language as this would be a “must” for KiCad to be seen as a professional tool.

Personally I do not use this part, maybe I should not write this at all, however.

Why would such a database have to be in KiCad itself? Why not a sort of BOM script, that converts the parts list to any sort of database back-end you prefer. It also does not seem too difficult to extract schematic symbol data with such a script and build automated part libraries from that for “preferred components”. The amount of work for such an interface is peanuts compared to the total effort in KiCad and it’s symbol and footprint libraries.

If it’s of any use, I can spend some time to find some links to this database related stuff.

craftyjon · May 10, 2020, 7:22pm

The database itself doesn’t have to be part of KiCad; KiCad just needs a way of talking to arbitrary outside data sources to get rows of symbol, footprint, and properties to automatically populate. It’s not sufficient to do it as a post-processing step if you want to match the workflow of the commercial tools – you need to be able to do an “update from database” operation inside KiCad to pull any changes to properties, symbol/footprint assignment, etc.

paulvdh · May 10, 2020, 7:27pm

So it’s 2 scripts, maybe a few more. Scripts for:

Exporting database info from KiCad to an external database.
Updating KiCad projects from the external database.
Combining KiCad libs and the Database to build libraries of preferred parts.
More ???

To me this looks like 50kiB of Python.

uschmelmer · May 11, 2020, 5:56am

@paulvdh i’m sorry if i insulted somebody.
It was just an example how a Tier 1 EDA Suite handles stuff.
And i think most of the stuff could be done via script.
But for that i think it’s mandatory to seperate symbol name from the value.
Than you have the possibility to bring every of your parts into kicad and still have easy readable schematics and layout printouts, eg for maintainance.
Having every part inside the schematic is quite important for an schematic engineers workflow in huge projects. Because then you dont get distracted by component selection. For example, do i have to use 1k2 (E24) or 1k21 (E48) resistor? I need 10u 25V, are these available in 0402, 0603, 0805 package? If you can define these once in the kicad library, you dont have to worry later on. And if you can assign unique names also the scriping stuff will be quite sufficient.

pointhi · May 11, 2020, 10:31am

I think a clean solution would be simply that python plugins can expose custom library “endpoints”. When KiCad asks “give me symbol mylib:A” it returns a lib object which the python plugin can generate on-the-fly. This allows everyone to write the library integration they need and interface with their systems.

paulvdh · May 11, 2020, 11:08am

No need to apologize, I have no Idea why you think I could feel insulted.
I just noted that this “database” thing is apparently important for a substantial group of people, and that it is a comparatively small effort to write an interface between KiCad and such an database.
I even find it likely that such tools already partly exist.
BOM tools are very near to export to a Database.
Some of the BOM tools integrate with spreadsheets.
Schematic symbols have custom fields for user defined data.
KiCost was made to get info from the web and combine with KiCad.
And there probably are more tools in this direction.

A custom field could for example be an UUID that defines a particular component class (0805 resistor, 0.05%, high stability) and the python scipt combines that with the value field to a completely specified symbol, and it complains if that value does not exist in that component type.

The main problem I see is the specifications of what these scripts should do exactly. Everybody seems to have their own and very specific demands of how they “need” it to work.

It’s also being made more complicated because (as I understand it) KiCad is and wants to remain neutral. So no order codes from a specific manufacturer in KiCad, or tools that rely on specific websites.

Resistors are a good example, as they come in many sizes, qualities, values etc. What do you think of this workflow:

Draw a schematic with default resistors (not even a footprint).
Extract data from KiCad schematic, and compare with external database.
Select specific component size / series / value etc in the database.
Port all changes into KiCad schematic.

Another workflow would be to have KiCad libraries generated from database info, and then use only the preferred parts from that database.
New parts would then fist be (pre-modified) and extracted from KiCad, and the fields and graphic lines of schematic symbols would then get copied into the external database.

I see at least 2 ways of such a system getting started.

First, someone can take up the scepter and implement exactly what they want for themself, and make it public. (It may exist already, as some companies internal secret sause)
Some of the people who want such an database put their heads together to find some common ground and make a concise set of specifications that can be implemented by a software guy.

A common problem with Open Source program is to have a good lead in which direction to go. KiCad is doing marvelous in this regard, thanks to a small group of motivated and wise people who make the right decisions.

I use Open Source software exclusively, to the point that I rather donate money to an Open Source project, then pay less for a commercial program that works twice as good. And over the years I’ve seen quite some open source projects that get get abandoned or hump along without really getting anywhere.

Having some good well written specifications of how such a database interface should interact with KiCad would be a mayor step forward and increase it’s chance of getting implemented. Some people do this for fun, and if they see such a document they think “Hey, I can do that in a week”, and then 2 months later it works…

uschmelmer · May 11, 2020, 12:40pm

Imagine some big project, really big: Intel Xeon Class computing module. 1000ths (!) of components. I have to select fully specified/atom symbols because right at this moment i am into this part of the schematic. On the other hand, layout is done simulteneously with schematic work (by another person), so footprints must exist.

I agree that external database interaction can be done via scripts.

But again i think having a decouling from the true symbol name and and its fields makes it quite easier and more clean.
I also wonder why then there are different places in the file format for symbolname and symbol value?

@stambaughw Btw, i just pulled the latest nightly and wondered about the structure of symbol libs. I tought it would be similar to the footprint libraries where a library is made by a directory and and every component resides in an extra file. This would be a lot easier if you work with multiple people on the same library

paulvdh · May 11, 2020, 12:54pm

I was just thinking out loud.
I have no interest myself in such a database, neither am I proficient enough with programming to write such scripts.

If you’re interested int trying to get something like this from the ground, then first search this forum and EEVblog, make some summary in a new thread, and possibly send some PM’s to others who seem interested in this to invite them to discuss some implementation for this.
There are already multiple threads about these databases both here on the KiCad forum and in the KiCad section on the EEVblog forum.

Also seems this thread has gone too far from it’s origin. Time to split it off?

stambaughw · May 11, 2020, 1:34pm

This is not practical because of inheritance. If you create a separate file for each symbol, inherited symbols would require copying multiple files to ensure the entire symbol can be constructed so I opted for multiple symbols per file to ensure the entire symbol is contained in one file. Maybe once the full inheritance is in place this will be easier to understand why I made this decision.

uschmelmer · May 11, 2020, 1:40pm

Ah ok, that makes it quite clear. Thanks for the quick clarification.
May i ask what the problem with different symbolname and value for symbol look up is? because there are two drifferent fields for symbol itself and the value:

(symbol "resistor:R_1K5_1%_0603_100PPM" (pin_numbers hide) (pin_names (offset 0))
(property "Reference" "R" (id 0) (at 2.032 0 90))
(property "Value" "R_1K5_1%_0603_100PPM" (id 1) (at 0 0 90))

why is look up not done by symbol field?

rrascher · May 11, 2020, 1:56pm

@uschmelmer

That’s was my question all along.
@stambaughw Maybe you could explain why there are two different fields, if they need to be the same
(at least in the symbol library).

stambaughw · May 11, 2020, 2:21pm

There is no technical reason. All of the internal data structures in the schematic and library editor code base are currently predicated on this behavior. Changing this would most certainly break the legacy symbol library file format which is out of the question. Implementing this would require deprecating writing to the legacy symbol library file format and making all legacy symbol libraries in the symbol library table read only. I don’t think this would be a trivial change in the code base. At some point this will happen but it will take some time. Once this is done, the naming behavior can be changed.

eeintech · May 20, 2020, 2:37pm

@stambaughw After checking the new format in the nightly build, I do have few questions.

If one wants to separate the “visual representation” of the symbol (managed by symbol editor) from the properties (assuming it is managed by eeschema library loader), I’m able to load the following .kicad_sch file just fine:

(kicad_sch (version 20200512) (host eeschema "5.99.0-unknown-4efed4b~101~ubuntu18.04.1")

 (page 1 1)

 (paper "A4")

 (lib_symbols
   (symbol "CAPACITOR_CERAMIC" (pin_numbers hide) (pin_names (offset 0) hide)
     (polyline
       (pts (xy -1.27 0) (xy -1.016 0))
     )
     (polyline
       (pts (xy -0.889 1.905) (xy -0.889 -1.905))
       (stroke (width 0.254))
     )
     (polyline
       (pts (xy 0.889 1.905) (xy 0.889 -1.905))
       (stroke (width 0.254))
     )
     (polyline
       (pts (xy 1.27 0) (xy 1.016 0))
     )
     (polyline
       (pts (xy 1.905 0) (xy 2.54 0))
     )
     (pin passive line (at -3.81 0 0) (length 2.54) (name "1") (number "1"))
     (pin passive line (at 3.81 0 180) (length 2.54) (name "2") (number "2"))
   )
 )

 (symbol (lib_id "CAPACITOR_CERAMIC") (at 77.47 85.09 270) (unit 1)
   (uuid "00000000-0000-0000-0000-00005e2185db")
   (property "Reference" "C2" (id 0) (at 80.137 82.7786 90)
     (effects (font (size 1.524 1.524)) (justify left))
   )
   (property "Value" "C1608X8R1E104K080AA" (id 1) (at 66.04 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Footprint" "Capacitors:C0603" (id 2) (at 50.8 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Datasheet" "https://product.tdk.com/info/en/catalog/spec/mlccspec_commercial_hightemp_en.pdf" (id 3) (at 48.26 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Supplier" "Digikey" (id 4) (at 63.5 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Supplier Part Number" "445-2500-1-ND" (id 5) (at 60.96 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Manufacturer" "TDK Corporation" (id 6) (at 58.42 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Manufacturer Part Number" "C1608X8R1E104K080AA" (id 7) (at 55.88 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Description" "CAP CER 0.1UF 25V X8R 0603" (id 8) (at 53.34 85.09 0)
     (effects (font (size 1.524 1.524)) hide)
   )
   (property "Capacitance (Farad)" "100nF" (id 9) (at 80.137 85.2678 90)
     (effects (font (size 1.27 1.27)) (justify left))
   )
   (property "Tolerance (%)" "±10%" (id 10) (at 73.66 96.52 0)
     (effects (font (size 1.27 1.27)) hide)
   )
   (property "Voltage Rated (Volt)" "25V" (id 11) (at 80.137 87.5792 90)
     (effects (font (size 1.27 1.27)) (justify left))
   )
 )
)

Is this the right approach?

Now, if one wants the properties of this capacitor to also be coming from the symbol editor / pre-loaded before it is placed in eeschema, I am trying to extend the “CAPACITOR_CERAMIC” symbol with properties:

(kicad_sch (version 20200512) (host eeschema "5.99.0-unknown-4efed4b~101~ubuntu18.04.1")

  (page 1 1)

  (paper "A4")

  (lib_symbols
    (symbol "CAPACITOR_CERAMIC" (pin_numbers hide) (pin_names (offset 0) hide)
      (polyline
        (pts (xy -1.27 0) (xy -1.016 0))
      )
      (polyline
        (pts (xy -0.889 1.905) (xy -0.889 -1.905))
        (stroke (width 0.254))
      )
      (polyline
        (pts (xy 0.889 1.905) (xy 0.889 -1.905))
        (stroke (width 0.254))
      )
      (polyline
        (pts (xy 1.27 0) (xy 1.016 0))
      )
      (polyline
        (pts (xy 1.905 0) (xy 2.54 0))
      )
      (pin passive line (at -3.81 0 0) (length 2.54) (name "1") (number "1"))
      (pin passive line (at 3.81 0 180) (length 2.54) (name "2") (number "2"))
    )
    (symbol "C1608X8R1E104K080AA" (extends "CAPACITOR_CERAMIC")
      (property "Reference" "C" (id 0) (at 0 4.064 0)
        (effects (font (size 1.524 1.524)))
      )
      (property "Value" "C1608X8R1E104K080AA" (id 1) (at 0 -11.43 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Footprint" "Capacitors:C0603" (id 2) (at 0 -26.67 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Datasheet" "https://product.tdk.com/info/en/catalog/spec/mlccspec_commercial_hightemp_en.pdf" (id 3) (at 0 -29.21 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Supplier" "Digikey" (id 4) (at 0 -13.97 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Supplier Part Number" "445-2500-1-ND" (id 5) (at 0 -16.51 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Manufacturer" "TDK Corporation" (id 6) (at 0 -19.05 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Manufacturer Part Number" "C1608X8R1E104K080AA" (id 7) (at 0 -21.59 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Description" "CAP CER 0.1UF 25V X8R 0603" (id 8) (at 0 -24.13 0)
        (effects (font (size 1.524 1.524)) hide)
      )
      (property "Capacitance (Farad)" "100nF" (id 9) (at 0 -3.81 0))
      (property "Tolerance (%)" "±10%" (id 10) (at 11.43 -3.81 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "Voltage Rated (Volt)" "25V" (id 11) (at 0 -6.35 0))
      (property "ki_keywords" "100nF 25V 0603" (id 12) (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
      (property "ki_description" "CAP CER 0.1UF 25V X8R 0603" (id 13) (at 0 0 0)
        (effects (font (size 1.27 1.27)) hide)
      )
    )
  )

  (symbol (lib_id "C1608X8R1E104K080AA") (at 77.47 85.09 270) (unit 1)
    (uuid "00000000-0000-0000-0000-00005e2185db")
    (property "Reference" "C2" (id 0) (at 80.137 82.7786 90)
      (effects (font (size 1.524 1.524)) (justify left))
    )
  )
)

The symbol lib reference would only overload the “Reference” field, right?

Unfortunately, I’m getting this error:

Error loading schematic file …
No parent for extended symbol CAPACITOR_CERAMIC in file …
line: 29
offset: 45

How would one manage this?

Thanks!

pedro · May 22, 2020, 2:53pm

Hi @stambaughw Wayne,

Some months ago I opened a feature request about the kicad_wks to be inserted in the new schematic/layout file format.

When we send or publish a project, the worksheet can be easily forgotten and lost for future distributions.

Is there a chance in V6 that the worksheet is embedded in the kicad_sch and kicad_pcb files the same way as the symbols and footprints? Thanks.

SembazuruCDE · May 22, 2020, 3:24pm

For reference, here is the issue in gitlab (I just thumbs upped it):

stambaughw · May 22, 2020, 4:08pm

@eeintech Inheritance cannot be used in the library symbols embedded in the schematic. The “extends” keyword is causing the failure. Inheritance can only be used in symbol libraries.

Please do not use “eeschema” as the host name for hand edited and script generated schematics. Change it to “hand-edited-by-me”, “my-leet-hacker-script”, or something similar. The “eeschema” token is reserved for by Eeschema formatter and is ignored by the parser. This way when you file bug report with your broken file, it is obvious to developers that this it was not generated by Eeschema. This applies to all KiCad file formats. This has caused countless hours of wasted KiCad developer time only to discover that it was due to files generated outside KiCad.

stambaughw · May 22, 2020, 4:12pm

Possibly, but doubtful. The are a lot of higher priorities for V6 so most likely this wont happen until V7 development.