Before I would resort to web scrapers again, I would want to see someone scrape mouser.com successfully for a week or so. They’re using Distil and I haven’t seen any people publicly claiming success at scraping of Distil-protected sites. And if you could, how long would that solution work? And how much effort are you willing to invest keeping your scrapers updated and running? Is that the best use of your resources?
I would rather have a service like Partinfo use their time/energy to get whatever info they can from public APIs, and use that to deliver value to the community. A service that provides part data without a great deal of effort will attract utility creators. More/better utilities will increase the size of the community. Eventually, distributors might give Partinfo increased access because they don’t want to be left out of that community.
Oh man, I just researched a bit. I thought Farnell was bad. “They transparently inject script tags that do fingerprinting, and they obfuscate the code that does so.” It’s funny because we are just trying to buy stuff from them!
What I like in PartInfo is that it may possible work on this issue https://github.com/xesscorp/KiCost/issues/17
and make possible an scrape / API for generic resistors just given the value + package + …
(use the value and packge from schematic / layout and give additional information, such: tolerance, material on KiCost GUI).
Yes! The goal for Electro Grammar, which is used by Partinfo, is to be able to give it text that describes any generic component. It only works for surface mount resistors, capacitors and LEDs at the moment but I should be able to work on it some more soon. David Craven already did a lot of work to compile transistors and diodes on an unmerged branch.
I could learn some JS to help. Language logical are almost the same in all “sequencial language”, if the community decide that is the better option centralize. I just may not be so helpful in specific configuration / aspects.
One curiosity: if I start I localhost, PartInfo application, this will make possible KiCost access it to scrape? @kasbah (just being careful in case of server down)
Yeah, use the dev branch and cp config.js.in config.js and add your Octpart API key. You probably want to just disable the Element14 API for now (it’s only used to get more accurate stock information). I changed it so if you don’t add the keys it won’t try and use the element14 API.
You will also need the Redis in-memory persistent database (used for caching). If you want to clear your local cache you can do redis-cli flushall.
I see that kitspace can provide the parts that match with some description. This could be interesting and KiCost, cold provide some high level (saying in programming language code) that select one of than by price / manufacture name or other characteristics in the case of https://github.com/xesscorp/KiCost/issues/17 .
I did check how to interact with and we will need some concept prove.
But first of this new feature, it important to restore the KiCost capabilities.
I’ll try and take a look when I have time. Already pretty busy these days unfortunately. Just skimming it seems like we’ll need to add some bits and pieces and adapt Partinfo as well.
Using GraphQL should actually simplify the querying process as it allows each user to only request what they need in a really granular fashion. E.g. “Farnell prices in Canadian dollars”. As we build out Partinfo we can make use of this and optimize what API/scraping that is actually called upon.
The quickest solution to keep KiCost running for now though may be to just create an Octopart proxy with my API key added.
In related news PCBWay donated $100 to opencollective.com/kitspace so we are a bit closer to our annual estimated budget to pay for the Octopart API.
Interesting, get the price in the correct currency was something that we was implementing at KiCost.
I don’t kown how configure this proxy, but with was just send the request to opencollective.com/kitspace instead www.octopart.com. It will be interesting to to keep the KiCost users ON during the correct development of our API.
Think that API should have:
get price breaks, available quantity, stock code of each distributor (when I search by manufacture part number), link for each component distributor page
Think that I would like at the API:
define the currency, datasheet link, image link, cycle life (in production / obsolete)
get the package/footprint, value, tolerance… (I am planning to use for validation at KiCost and I think by ElectroPart you may already have)
When I find sometime I will start to write at GitHub, at least the issue and develop plans.
Hi, are there other cost providers?
Maybe we should use digikey and rs to get prices.
So I also feel that they would help to provide an
API as they are interested in sales. Small enterprises
are using EAGLE and as EAGLE turned more evil
(subscription only) KICAD is a good alternative
when it became stable an reliable. So EAGLE designs
can be imported and stability improved with 5.2.1 version
so having KICAD with a KICOST which directly spiders
some major distributors would be the way. So we are
not dependent on a single vendor like octopart
which has been aquired by evil altium, as altium does
not want to release their file format description,
we should not be dependent on a single vendor!
It would be like writing an application for facebook
when they change their API or shut down their services,
all the work is void.
Major distributors won’t shut down their online shops.
So if one distri changes his shop the others won’t do
it at the same time.
So go for the distributors and not for octopart which
was aquired by evil altium.
Cheers and Thanks to all coders and librarians for KICAD!
@xennonflash
Next version of KiCost add support to PartInfo - KiSpace API that in some point will make KiCost be able also to automatic select resistors and capacitors without provide the part number.
It will be also able to support additional APIs for each distributors web site. If someone want to help, provide a package with distributors/api_DISEREDDISTRIBUTOR.py the development team will be glad to merge it to the main branch.
For now KiCost team are composed just by one person + the author and we need people to provide fix and report issues. Mainly the related with the GUI and Windows behavior, due to this there were no recent releases.
