Just came across this thread and maybe we can help out a bit.
As mentioned before, our Precious Parts offering makes use of DigiKey and Farnells APIs natively. We’d be happy to provide open access to our part search free of charge.
Does that sound like a plan? If so we’d have to build an open API for this.
As we do not make our money with advertising we’ve no plans to charge or (like Octopart) lock down the API.
You will need some dependence on something at some point. Just keep it in mind as a tool if you run into scraping issues. I hope that a Partinfo fallback in your lib won’t cause problems for making use of your lib in Partinfo. 
@aisler, very generous! As you know I am waiting to make use of your API in Partinfo but I didn’t want to announce anything without checking with you first.
+1 from my side…
the best option to avoid a distributor banning is acting with a client emulating a browser behavior… No heavy mass requests from a single IP or server
IMO standard API is a dead route, because of the dependency on distributors inclination.
Great, but I don’t know how to really stay out robot detection / use browser emulation.
I just concerned here: I can’t say to users of KiCost (for example) that it just can use installing Chrome or Linux OS (because of the package used for browser emulation).
So browser emulation may be the way, since it is not strict to: one browser, one OS or a lot of user configurations.
Sure, maybe you can emulate a browser well enough so you are not flagged. I think actually using a browser is the safest bet though. Chrome Headless is available for all platforms, doesn’t need configuration and shouldn’t interfere with the users default browser at all. Did you see this note in the Pyppeteer docs?
Note : When you run pyppeteer first time, it downloads a recent version of Chromium (~100MB). If you don’t prefer this behavior, run
pyppeteer-installcommand before running scripts which uses pyppeteer.
Before I would resort to web scrapers again, I would want to see someone scrape mouser.com successfully for a week or so. They’re using Distil and I haven’t seen any people publicly claiming success at scraping of Distil-protected sites. And if you could, how long would that solution work? And how much effort are you willing to invest keeping your scrapers updated and running? Is that the best use of your resources?
I would rather have a service like Partinfo use their time/energy to get whatever info they can from public APIs, and use that to deliver value to the community. A service that provides part data without a great deal of effort will attract utility creators. More/better utilities will increase the size of the community. Eventually, distributors might give Partinfo increased access because they don’t want to be left out of that community.
this seems a good start
https://www.seleniumhq.org/projects/webdriver/
and this
https://code.google.com/archive/p/pyv8/
Oh man, I just researched a bit. I thought Farnell was bad. “They transparently inject script tags that do fingerprinting, and they obfuscate the code that does so.” It’s funny because we are just trying to buy stuff from them! 
What I like in PartInfo is that it may possible work on this issue https://github.com/xesscorp/KiCost/issues/17
and make possible an scrape / API for generic resistors just given the value + package + …
(use the value and packge from schematic / layout and give additional information, such: tolerance, material on KiCost GUI).
Yes! The goal for Electro Grammar, which is used by Partinfo, is to be able to give it text that describes any generic component. It only works for surface mount resistors, capacitors and LEDs at the moment but I should be able to work on it some more soon. David Craven already did a lot of work to compile transistors and diodes on an unmerged branch.
I could learn some JS to help. Language logical are almost the same in all “sequencial language”, if the community decide that is the better option centralize. I just may not be so helpful in specific configuration / aspects.
One curiosity: if I start I localhost, PartInfo application, this will make possible KiCost access it to scrape? @kasbah (just being careful in case of server down)
Yeah, use the dev branch and cp config.js.in config.js and add your Octpart API key. You probably want to just disable the Element14 API for now (it’s only used to get more accurate stock information). I changed it so if you don’t add the keys it won’t try and use the element14 API.
You will also need the Redis in-memory persistent database (used for caching). If you want to clear your local cache you can do redis-cli flushall.
That said I am happy for you to use the dev-partinfo.kitspace.org/graphql endpoint for testing, i think the traffic will be ok.
Edit: I added some instructions to the repo and made it so that not giving any element14 API keys will simply not try and access that API.
I see that kitspace can provide the parts that match with some description. This could be interesting and KiCost, cold provide some high level (saying in programming language code) that select one of than by price / manufacture name or other characteristics in the case of https://github.com/xesscorp/KiCost/issues/17 .
I did check how to interact with and we will need some concept prove.
But first of this new feature, it important to restore the KiCost capabilities.
Yeah, the BOM Builder can do that (kitspace.org is what I call Kitspace) but it’s actually built into Partinfo and the BOM Builder is using Partinfo.
If you run into something Partinfo can’t do that’s required to restore current KiCost functionality, let me know please and we can see how to add it.
@kasbah, could you help to modify https://github.com/xesscorp/KiCost/blob/master/kicost/distributors/dist_octopart.py to get data from kispace?
I’ll try and take a look when I have time. Already pretty busy these days unfortunately. Just skimming it seems like we’ll need to add some bits and pieces and adapt Partinfo as well.
Using GraphQL should actually simplify the querying process as it allows each user to only request what they need in a really granular fashion. E.g. “Farnell prices in Canadian dollars”. As we build out Partinfo we can make use of this and optimize what API/scraping that is actually called upon.
The quickest solution to keep KiCost running for now though may be to just create an Octopart proxy with my API key added.
In related news PCBWay donated $100 to opencollective.com/kitspace so we are a bit closer to our annual estimated budget to pay for the Octopart API.
Interesting, get the price in the correct currency was something that we was implementing at KiCost.
I don’t kown how configure this proxy, but with was just send the request to opencollective.com/kitspace instead www.octopart.com. It will be interesting to to keep the KiCost users ON during the correct development of our API.
Think that API should have:
get price breaks, available quantity, stock code of each distributor (when I search by manufacture part number), link for each component distributor page
Think that I would like at the API:
define the currency, datasheet link, image link, cycle life (in production / obsolete)
get the package/footprint, value, tolerance… (I am planning to use for validation at KiCost and I think by ElectroPart you may already have)
When I find sometime I will start to write at GitHub, at least the issue and develop plans.
Hi, are there other cost providers?
Maybe we should use digikey and rs to get prices.
So I also feel that they would help to provide an
API as they are interested in sales. Small enterprises
are using EAGLE and as EAGLE turned more evil
(subscription only) KICAD is a good alternative
when it became stable an reliable. So EAGLE designs
can be imported and stability improved with 5.2.1 version
so having KICAD with a KICOST which directly spiders
some major distributors would be the way. So we are
not dependent on a single vendor like octopart
which has been aquired by evil altium, as altium does
not want to release their file format description,
we should not be dependent on a single vendor!
It would be like writing an application for facebook
when they change their API or shut down their services,
all the work is void.
Major distributors won’t shut down their online shops.
So if one distri changes his shop the others won’t do
it at the same time.
So go for the distributors and not for octopart which
was aquired by evil altium.
Cheers and Thanks to all coders and librarians for KICAD!
@xennonflash
Next version of KiCost add support to PartInfo - KiSpace API that in some point will make KiCost be able also to automatic select resistors and capacitors without provide the part number.
It will be also able to support additional APIs for each distributors web site. If someone want to help, provide a package with distributors/api_DISEREDDISTRIBUTOR.py the development team will be glad to merge it to the main branch.
For now KiCost team are composed just by one person + the author and we need people to provide fix and report issues. Mainly the related with the GUI and Windows behavior, due to this there were no recent releases.
The old announcement
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.