Why is generation of the BOM such a nightmare?

Schematic
1

The plugins don’t seem to work as advertised (in any case, as I understand the advertisement).

For example, from eeschema, I go menu: Tools → Generate Bill of Materials.... I have the plugin bom_csv_grouped_by_value. Except that it does not group by value: seems to group by the combination (value,footprint); or maybe (value,LibPart); or maybe (value,footprint,LibPart) ??

When you think about it, it wouldn’t make sense to group by the field Value alone; you could have resistors that are all 1k (so, the value is 1k), but then some are 0603, some are 1206, some are TO-220, etc.

However, one obvious problem is: one could have the same part on different footprints. (the simplest example is perhaps for SMT passives; one may have the three different IPC-7351 recommended footprints: for low-, medium-, and high-density placements)

For all my parts, I have two custom columns: MPN, and Digikey part number; whenever I generate a BOM, I have to then open the CSV in a spreadsheet software, sort by Digikey part number, and then manually identify the rows that should be merged (sorting by that column just makes it easier to identify the rows that should be merged).

I saw that there are other plugins, but they don’t seem to work well. If I choose either bom_html_grouped_by_value or bom_html_with_advanced_grouping, they just won’t output the columns Digikey and MPN. (bom_csv_group_by_value simply outputs them; I don’t recall configuring anything, and the Generate BOM dialog does not have any configuration options).

For that matter, quite ironically, the plugin “with advanced grouping”, well, I don’t know what’s advanced about it; it does not offer me any configuration, and it seems to just do an equally poor job as the html_group_by_value.

What am I missing? I would really like to avoid any tools that I need to get from some random person’s github page or something like that (don’t get me wrong: kudos to those volunteers that put the effort to contribute to KiCAD… I’m just, what can I say, I strongly prefer to avoid any of those alternatives).

2

To clarify a bit … I guess as usual in my posts, with all the ranting, it can be difficult to figure out what I’m asking :slight_smile:

I would very much like a BOM generation that allows me to:

  • Select which fields I want (extra fields are not the problem: it’s very easy to edit with LibreOffice and remove columns; but I would like to be able to ensure that a certain field will appear on the BOM)
  • Group by a field or combination of fields selected by the user (for example, I would like to group by Digikey part number, which then guarantees that there won’t be multiple lines that indicate the same actual part).
3

I have been doing that way since over 20 years (changed during moving to KiCad recently). We decided that our standard is 0603 so value 1k = 1k 0603. In rare cases when we used different sizes we just added that info to value so 1k_4 was 1k 0402.

I had also that problem until I start to use KiCad. Looking around how things work here I organised everything that way that I get correctly merged all elements from KiCad (in csv used then to add the full element specifications in spreadsheet).

I am using bom_csv_grouped_by_value_with_fp. But I have it modified a little. I don’t know Python but I got some help here so I managed to make needed by me modifications and than later I have done some more by trying to understand the source and some experimenting.

4

Good points, Piotr!

I’ve thought about standardizing the footprints, as you describe.

However, the plugin bom_csv_group_by_value does not group by value. My CSV output shows multiple different lines with the same value in the column “Value”. Example (copy-n-pasted from one of my CSV outputs — erasing rows to leave only capacitors, showing some of the repeats):

"Item","Qty","Reference(s)","Value","LibPart","Footprint","Datasheet","Alternative Digikey","Alternative MPN","Digikey","MPN"
"3","4","C1, C2, C3, C4","0.1uF / 25V","local:C_decoupling_402","local:C_decoupling_402-axial","~","","","399-14148-1-ND","C0402C104K3RAC7867"
"4","3","C5, C6, CM3","0.1uF / 16V","Device:C_Small","Capacitor_SMD:C_0402_1005Metric","~","","","399-3521-1-ND","C0402C104K4RACTU"
"5","10","C7, C8, C10, C11, CIN5, CM1, CMB3, CP1, CP3, CP5","0.1uF / 50V","Device:C_Small","Capacitor_SMD:C_0603_1608Metric","~","","","399-5089-1-ND","C0603C104K5RACTU"
"6","2","C9, C18","0.1uF/100V","Device:C_Small","Capacitor_SMD:C_1206_3216Metric","~","","","445-172694-1-ND","CGA5L1C0G2A104J160AC"
"7","4","C12, C13, C14, C15","150pF/2kV","Device:C","Capacitor_SMD:C_1206_3216Metric","~","","","399-13500-1-ND","C1206C151JGGAC7800"
"8","59","C16, C17, C23, C24, C25, C26, C27, C28, C29, C30, C31, C32, C33, C34, C35, C36, C37, C38, C39, C40, C41, C42, C43, C44, C45, C46, C47, C48, C49, C50, C51, C52, C53, C54, C55, C56, C57, C58, C59, C60, C61, C62, C63, C64, C65, C66, C67, C68, C69, C70, C71, C72, C73, C74, C75, C76, C77, C78, C79","0.1uF / 16V","Device:C_Small","local:C_0402","~","","","399-3521-1-ND","C0402C104K4RACTU"
"9","2","C19, C21","0.47uF","Device:C_Small","local:C_0603-smallest","~","","","399-11934-1-ND","C0603C474K3RAC7867"
"10","2","C20, C22","22uF","Device:C_Small","Capacitor_SMD:C_1206_3216Metric","~","","","587-3425-1-ND","LMK316AB7226KL-TR"
"11","2","C80, C81","1nF/NPO","Device:C_Small","Capacitor_SMD:C_1206_3216Metric","~","","","DNP","DNP"
"12","3","CA10, CA19, CA37","C_decoupling_402_gnd_vias","local:C_decoupling_402_gnd_vias","local:C_0402","~","","","478-10043-1-ND","04025C104KAT2A"
"13","1","CA28","0.1uF","local:C_decoupling_402_gnd_vias","local:C_0402","~","","","478-10043-1-ND","04025C104KAT2A"
"14","5","CA49, CA53, CA59, CA63, CCM1","C_decoupling_402_gnd_via","local:C_decoupling_402_gnd_via","local:C_0402","~","","","478-10043-1-ND","04025C104KAT2A"
"15","5","CAN1, CLKA3, CLKA4, GND5, VDD5","TestPoint","Connector:TestPoint","local:MicroTestPoint","~","","","",""
"16","1","CB3","47uF","Device:CP1","Capacitor_SMD:CP_Elec_5x5.9","~","","","565-4532-1-ND","APXG200ARA470ME61G"
"17","2","CC1, CC2","0.5pF","Device:C_Small","local:C_0402","~","","","399-11253-1-ND","CBR04C508A3GAC"
"18","1","CC3","0.1uF / 25V","local:C_decoupling_402","local:C_decoupling_402","~","","","399-14148-1-ND","C0402C104K3RAC7867"
"19","1","CC5","0.1uF / 25V","local:C_decoupling_402_gnd_via","local:C_decoupling_402","~","","","399-14148-1-ND","C0402C104K3RAC7867"
"20","16","CE1, CE4, CE12, CE16, CE18, CE20, CE25, CE30, CE36, CE40, CE42, CE46, CE54, CE59, CM80, CM81","0.1uF / 16V","local:C_decoupling_402","local:C_0402","~","","","399-3521-1-ND","C0402C104K4RACTU"

You can see multiple lines where Value is 0.1uF / 16V (C0402C104K4RACTU), multiple lines with Value 0.1uF / 25V (C0402C104K3RAC7867), etc.

These weird custom footprints I created, one could argue that those were a bad idea to begin with (I would probably not argue too much; in hindsight, it doesn’t look like such a good idea as it did when I decided to create them), but that does not negate the fact that bom_csv_group_by_value does not work as advertised (the “group by value” part)

Am I missing something?

5

Csv is a very simple file “format” which can be opened in any spreadsheet application. There you can manipulate, group and order the rows automatically as long as you know how to use that application.

Anyways, I agree that creating a BOM is more complicated than it needs to be. If I can use a spreadsheet application, all I need is a csv file with all available fields. Now I have to find a script which does that job (the default one doesn’t export all fields). Why not just have it as default? It’s easy to delete columns and order rows in a spreadsheet application. And the “Run command” panel and “Command line” are gibberish for those who don’t know about certain technical details in advance – actually I’m a programmer and know a thing or two about command line, but even I find difficult to understand what happens.

If I would like to prepare the BOM further in KiCad I would rather appreciate a GUI with options: select fields and ordering etc. Save them as named presets. The need to have a script for each different option set feels just stupid. I can understand the current way is the most versatile and generic for everything, but it’s unnecessarily user-unfriendly for most needs.

And think about the script names… for example

  • bom_sorted_by_ref
  • bom_csv_grouped_by_value
  • bom_csv_sorted_by_ref
  • bom_html_grouped_by_value

Doesn’t this tell immediately that there’s something conceptually very wrong with this system? Of course these should be options:

  • csv/html/xml etc.
  • grouped by
  • sorted by
  • etc.
1 Like
6

Totally agree - terrible BOM tool.

FYI - you can edit the desired .py file and customize it to suit your needs. Very easy - First copy one and renmame it. Then, edit… Add/delete fields, tweak the format…

If you review several of the bom py files, you may learn what to add/delete and how to format output - just takes some up-front effort. No need to become a Python master - just a few basic notions to understand…

If loading a CSV Bom into LibreOffice, the popup panel allows for Hiding Columns (Right-Click) before it loads into the spreadsheet…

And, if not bent on using the BOM from Eeschema and you have a PCB, PCBnew|File>Fabrication Output>BOM exports a default CSV of the PCB (Not the schematic). Of course, you could load the annotated/Net into the PCB then, export the BOM without bothering to arrange the footprints (for quickness)…

7

My skills with LibreOffice Calc (or MS Excel, for that matter) are barely above absolute beginner. For example, I haven’t the slightest idea of how to merge two (let alone any number of) BOM lines that should be merged. Example:

Lines:

Qty, Refdes, Value, MPN
2,"C1,C2","0.1uF / 16V", C104xx
4,"C10,C11,C12,C13","0.1uF / 16V",C104xx

Merge (grouping by MPN) into:

Qty, Refdes, Value, MPN
6,"C1,C2,C10,C11,C12,C13","0.1uF / 16V",C104xx

Any tips (if it is simple enough as to explain here, or just give me the pointers) would certainly be appreciated.

8

Curious that you suggest this !! (do you by any chance recognize my username and remember that I am a programmer / have experience in programming?)

At any rate … Although having a lot of experience in programming, I have to confess that I am pathologically phobic to Python… I would probably prefer to reverse-engineer (or lookup the docs on file formats) the .sch files, then write a program in C++ that reads the .sch files and just entirely build and output the BOM, rather than having to work with some Python program (really: I’m not kidding! Although I know that it wouldn’t make sense to work with the .sch files; better to use the “Individual Components” section of the BOM, as produced by KiCAD with the bom_csv_group_by_value plugin)

9

Oh, that’s what grouping means… You can see I have been reluctant to learn KiCad BOM. :slight_smile: I don’t think it’s easily possible to do in a spreadsheet without programming. But if you know python or visual basic you can script LibreOffice or Excel :smile:

10

Hahaha, fair enough!

I mean, I assume that that’s what group by refers to, based on the SQL notion of group by in SELECT statements. Actually, now that I remember, the Tools → Edit Symbol Fields... dialog also has a “Group by” option, and it is consistent with this behaviour.

11

I dislike Python. Period. To me, it’s not a real programming language.

But, I forced myself to learn just enough to make Kicad Plugins. It took 10 minutes.

I certainly don’t want to encourage you out of the realm of your comfort zone so, I conclude my input by saying (with respect to the below image) most any programmer of most any language (Java, C, Basic, Fortran…etc) will quickly recognize the commonality of Code elements/syntax of:
-Comments
-Import
-Try/Exception
-For
-Write
-get

Sure, specific syntax details can differ (such as ‘except’ versus ‘exception’) …

Screen Shot 2020-09-29 at 2.42.51 PM
Screen Shot 2020-09-29 at 2.42.51 PM1338×1600 959 KB

12

It might be easier to parse the intermediate XML file that eeschema generates as part of the BOM process. (This is actually what the other BOM plugins do, both the xsltproc and python BOM generators.) If you look at some of the other BOM plugins, particularly the “Command line:” field there is the “%I” which is expanded to the schematic name with .xml as the extension. This XML file is actually an XML based netlist. I would expect there are many C++ xml parsers out there, you may even already have a favorite that you’ve used in other projects. I threw together a simple 4-component project for something else, but it has a small xml file. Here is the schematic for reference:

2020-09-29 22_28_49-Eeschema — Test.sch _ — D__Libraries_MyDocuments_Dropbox_RemoteDevelopment_KiC
2020-09-29 22_28_49-Eeschema — Test.sch _ — D__Libraries_MyDocuments_Dropbox_RemoteDevelopment_KiC997×542 30.5 KB

And here is the xml file: Test.xml (4.5 KB)

(BTW, the InteractiveBOM plugin also uses that .xml file instead of parsing the .sch file to get the fields for each component that aren’t in the PCB data model.)

2 Likes
13

Yeah, if you are a programmer but can’t stand python, eeschema is not opinionated on this and doesn’t force you. All the built-in bom tool does is generate a netlist file in xml format that has all the components, their pins, custom fields, and nets connecting the pins. Then it launches an external process and passes the path to the xml to it (if you have %I in the command line). That process can be anything, you can write your own bom tool and group however you want in whatever language you want.

InteractiveBOM also understands .net files.

2 Likes
14

But .net files are not really created in the current kicad workflow. At least the xml file kind of makes sense as it is what normally drives BOM creation so is kind of reasonable to be needed by a BOM script even one running in pcbnew.

15

Yes, except most people have no idea what it is or how to get one because it is not explained anywhere. And it’s a pain to do in default configuration because bom tool actually requires you to select some bom script even if all you need is just an xml file.
But if I put in the docs “generate a netlist file” everyone knows what it is referring to.

16

Not trying to stop you here, but maybe you want to try to one of the other BOM plugins, from people that not unlike you, didn’t like the standard stuff.

17

I also don’t like python. However modifying few lines to get exactly the output I need was not that terrible.

Just choose the plugin that groups the way you like and modify the fields it outputs

18

May be.
I started when 4.0.7 was current. When I tried to update to 5.0 I found that I had to stop using ‘/’ in my element values. At that moment I don’t remember exactly what problems I noticed but I remember that I asked (not sure if here or at bug reports) and was told that it is because in v5 some library is used and it had to be such. But in v6 I will be once more able to use ‘/’ in my values.
Currently I have all values without ‘/’ in them.
But I never tried plugin you use - I only tried that … by_value_with_fp.

19

Not exactly for your question…
The plugin I use does that job for me (but I don’t use ‘/’ - may be it is a problem).
I remind that plugin I used wrongly sorted the references - so it happened for example C1,C3,C7,C4.
But when I read plugin I sow there the call to sort it - didn’t understood why it not works.
During my read I found that there are tools ready to eliminate elements I don’t wont in my BOM (like pads to solder wires). But those tolls were not used in plugin I used. Doing some things as a blind (don’t know Python) I added use of that tool in my plugin with side effect of sorting references being called twice. Each time the call written slightly differently (it is not in modified by me plugin but in something external used in it). The end effect is that I have references sorted correctly. I decided to not save computer time and left it doing that twice :slight_smile:
I reported it as a bug, but I am not skilled enough in bug system to find it. Those time they modified slightly the plugins (in the comparing equation used in sorting) but for me it looked that with no effect. I don’t know if something later was done with it.
If you need I can share my plugin file but it is based on source valid when v 4.0.7 or 5.0 was on top.

20

Although we are already in the off-topic realm, I feel compelled to reply/comment: what you say makes sense. For the longest time, in my mind that idea has been present (you know, that “at some point in time, I will just have to get over my profound dislike for Python and at least get familiar with the basics”) … The fact that it’s been years with that idea present, yet never finally picking it up is in part what makes me describe my “condition” as a pathological phobia :slightly_smiling_face: