Yes. There are a maximum of 30 active processes and each one is web-scraping for a single part on Digi-Key, then Mouser and then Newark. Parallelizing those scrapes might get some more speedup.
That’s certainly a possibility. I stuck with using the multiprocessing module since that’s in Python’s standard library. I figured that might lessen the number of installation and cross-platform problems. All the process spawning in KiCost is done at a single, small section of the code so it should be easy to experiment with other parallel processing modules.
Yeah, but you would still have to web-scrape every time to check and see if the part info in the cache was still valid. And actually, the spreadsheet that’s generated is kind of like a cache since it contains all the quantity-price information for each part at the time the scrape was done (that’s needed so you can change the production quantity within the sheet and still get an accurate, quantity-adjusted price).
Yeah, but you would still have to web-scrape every time to check and see if the part info in the cache was still valid. And actually, the spreadsheet that’s generated is kind of like a cache since it contains all the quantity-price information for each part at the time the scrape was done (that’s needed so you can change the production quantity within the sheet and still get an accurate, quantity-adjusted price).
I see, but over the course of some short period this should still be valid. i.e. I make a change to my design and want to see what the effect on the cost. Another thing is the price from a vender I would suspect would change slower than stock of the part.
OK, yeah, I can see your point. Python has modules for doing small, local databases so it might not be too hard to build this. Of course, that’s not something I’m planning to do anytime soon.
While we’re here @devbisme.
I didn’t see it mentioned anywhere and only had a short trial without success - but can KiCost work with it’s own output (xls) as input for scraping?
That’s what I did infer from the flowchart for KiCost, but I couldn’t find the option to use a modified xls as input for KiCost to get me an updated xls with the cost/part updated?
KiCost only accepts XML files output by the BOM tool of EEschema. Basically, the flow is like this:
Output the XML file for a schematic in EEschema.
Run KiCost on the XML file. It will scrape the distributor websites for the pricing & availability data. (The scraping is represented by the arrow from the online distributors to KiCost.)
KiCost will output an XLSX spreadsheet file that can be opened using Excel, Google Sheets, or LibreOffice Calc. One of the functions of the spreadsheet is creating a list of parts that can be cut-and-pasted into the online order forms for the distributors. (That’s the arrow from the spreadsheet program back to the online distributors.) The online order forms just get a formatted piece of text listing the catalog numbers and quantities for each part. They don’t get an actual XLS file so there’s no pathway for inputting such a file to KiCost.
I’m curious as to what function you thought would be enabled if KiCost could import its own XLSX file. That might be a new capability that could be supported.
On a separate note, let me extend my appreciation for all the detailed answers you’ve provided on this forum. I know that’s not easy work and it’s often thankless. I’m just a user (not a developer), but I want KiCad to succeed and the assistance you and many others provide is fundamental to that success. Thank you!
If you start fresh (newbie) and don’t know that automatic BOM generation with order-list ready output might be desirable (doh) all your components will lack those fields (or once you finally make the switch from global to local libs all the former globals will miss those fields). The first thing I did was using KiCost as a BOM to XLS converter - so to speak - for a somewhat ordered central collection of those values.
I just copy&pasted the manf/manf# fields into the spreadsheet - doing this in excel is way more comfortable vs. doing this in KiCAD.
This naturally means KiCost couldn’t really do it’s main job - scraping those distributors - so all the order relevant fields where not populated
If KiCost would had been able to take in it’s own output I could have had order-ready output after I had populated those fields in excel. So I had to manually copy&paste some areas of the spreadsheet and just went with DigiKey by using it’s BOM-upload facility and worked from there.
Another thing to consider for KiCost, is when you want to check some options for similar devices - adding a couple of rows in excel, populating them with manf# and letting KiCost do the work of finding prices for several distributors - would be very neat and fast.
A completely unrelated but pretty cool feature I can think of (which relates to the first point above):
It would be nice if KiCost would be able to take it’s output xls file and adding the manf/manf# fields to the local libraries, so one could use KiCost not just for costing an assembly, but also as a means to conveniently populate the local KiCAD libraries with the manf/manf# fields.
Might even pay off to add the option of custom fields via KiCost to have them added to KiCAD (like internal whatevers) or changing field values (it’s easier to rename devices when you can see all of them underneath each other vs. if you can only do one-by one in KiCAD).
But maybe this should be better done with a stand alone lib<>xls converter that works both ways?
Ha, right back at ya!
Your script/prog saved me a lot of confusing and error-prone manual copy&paste to get the right amount of devices for my boards. I’m sure I’d been able to come up with some sort of python script on my own, but KiCost is already doing this job - so big thanks for saving me a lot of work.
KiCost should be able to read a spreadsheet formatted similarly to the sheet it outputs, and use the information in that sheet (primarily the manf#) to create a new sheet with all the costing data in it.
You should be able to add new rows (or copy existing rows) in the spreadsheet, change the manf#s, and KiCost will create a new sheet with the additional pricing for the new part numbers.
Back-annotate the manf# (or even other columns) from the spreadsheet into the schematic and/or library.
If that’s an accurate reiteration of what you’ve described, I can enter those into the KiCost Github as enhancements. None of them seem particularly hard to do although I can’t say when I might get to them.
Correct, very concise and what I wanted to say.
Point 2) should be very useful for anyone and is being possible at the same moment as 1) is doable.
3) is just the icing with a cherry on top.
Good luck with it.
I haven’t looked at KiCost yet but am I right that it’s doing screen scraping to get pricing from suppliers? You may want to use octopart’s API instead. I’d think it would be easier:
The only thing is that I am in Australia and whenever I click one of the links I go to the distributor sites but get the AU version of those sites with pricing in AU$ while it appears KiCost scrapes costs in US$ so it is sometimes a bit confusing as what I am actually looking at.
Is there any way to ‘configure’ maybe the local country so I can select if I get the local distributor site and which currency I get the results in?
Regardless, can I suggest that the spreadsheet has the currency a bit more explicitly indicated somewhere? Like for example the total cost or unit cost label as in 'Total Cost (US$)" and 'Unit Cost (US$):"?
Other than that, great script. Absolutely love it.
Adding the US$ to the label is easy, but doesn’t solve the real problem you’re having.
KiCost really needs to be internationalized. That’s something I know very little about. Let me know if anyone has a pointer to some quick method for doing that.
Even these are internationalised - I use Malaysia versions of RS and Element14, priced in MYR.
For hobbyist or very small business, not requiring concurrent multi user write access, sqlite is lightweight and could even be distibuted with KiCad as it is in the public domain
I think this is a brilliant tool , with possibilities to grow into something biger
web -> SQL -> Kicad
And i think about components
you could basically automate all your component libraries creation
but i think mostly of caps and resistors that goes in long series of values
lets say you ask the tool to grab all the murata grm15** series capacitors from mouser.
push into sql and make parts from a template for symbol and footprint
but take all the values from mouser
Part-no (Value)
Real value (Value2)
tolerance
manufacturer
rating
dielectric
etc
etc and automatically generate the whole series of caps into your murata library
nice
the sql database becomes your real library
2 different tools, collect and generate
SQL in the middle
I was sourcing resistors from RS and there were significant price variation between brands at each value. The trouble was that for each value the cheapest was different. If you focus on one brand, your BOM cost will be inflated, so being lazy is punished
