How to integrate a database library?

Projects
1

I publish the largest open source database library for Altium and have multiple requests to see if I can do something to port it over to KiCad. I haven’t used KiCad before, so I don’t know what would be involved, or if it’s even possible.

An Altium Database Library (DbLib) works by having a database (in my case, hosting on Azure for now, soon our own dedicated server) which associates footprints to schematic symbols and contains all the parametric information for the part. In my case, I have a single table with all 30k parts, and then database views which break those down into categories (resistors, mlcc caps, ARM controllers, etc) - similar categories as you’d find on DIgiKey. Each part is associated at the moment with a DigiKey part, so when you export the BOM it has the DK link, SKU, etc for ordering the exact part you placed. It also knows the datasheet link and such so from Altium you can jump right to the datasheet for the part from the schematic view.

Altium uses STEP (AP214) 3D models, so every part has a full colour, correct-to-the-manufacturers-datasheet sized 3D models

From what I’ve been able to figure out, there isn’t a way to connect such a DbLib directly, even if there were KiCad compatible schematic and footprint libraries available. So, what would be my best approach to export all the data? I’m guessing I’ll have to write some software to translate the Altium SchLibs and PcbLibs, and generate a KiCad library from the database data?

Looking at the library docs, I think this is going to need to result in a massive number of files, such as ‘capacitor-mlcc-0603-murata’ or something like that? I’d still be likely to blow through the 250 limit really quickly on the passives though. When filtering through say capacitors in Altium i’ll group by component size, then value, then voltage - so I’d dig down through 0603 capacitors that are 0.1uf, and 25v - sorting by price to pick a part. Any suggestions on how such a thing my be accomplished with KiCad?

I feel my sch symbols all comply to the library requirements, if I can figure out how to translate them over from Altium. Footprint naming conventions will not meet requirements… oh well.

Any suggestions on how I might be able to proceed? Or alternatively - would there be any interest in building dblib functionality into KiCad (I’d need a lot of help with that)?

5 Likes
2

KiCad stores the footprint pointer and bom info directly in the symbol. This means you would need to combine the altium symbol (which i assume only holds graphical information plus connection points for wires) with the footprint info and bom info and generate a kicad symbol from that.

3d models in kicad are directly assigned to the footprint. This would mean something similar as with symbols might need to happen where you add the 3d model info directly to the footprint.

In the end this will likely mean that you need to run scripts generating kicad assets out of the database structure. This will of course mean that this will not update automatically without rerunning that script.

1 Like
3

There has been work done on:
https://duckduckgo.com/html?q=“Altium+to+KiCad”

All of KiCad’s file formats are pretty simple and readable ASCII, and as an Open Source project description of file formats is in the documentation:
http://www.kicad.org/help/file-formats/

http://bazaar.launchpad.net/~stambaughw/kicad/doc-read-only/download/head:/1115%4016bec504-3128-0410-b3e8-8e38c2123bca:trunk%252Fkicad-doc%252Fdoc%252Fhelp%252Ffile_formats%252Ffile_formats.pdf/file_formats.pdf

There are also a lot of side projects floating around KiCad and there are Python libraries for reading KiCad files and manipulating data, but I’ve never used those myself.

KiCad also supports STEP for 3D models.

1 Like
4

For footprints this is a good start: https://github.com/pointhi/kicad-footprint-generator

For symbols (and reading footprint data) the library utils repo has some options https://github.com/KiCad/kicad-library-utils

1 Like
5
6

Forgot to link the library: https://github.com/issus/altium-library :slight_smile:

2 Likes
7

Moderator comment:
I cannot see any hidden or withdrawn messages in this thread

8

The initial post from @issus disappeared for a while with a moderator flag - not sure why, anyhow it is now back OK.

9

Sounds good, so it’s just going to be a matter of decoding the alitum pcblib and schlib files, then I can generate the footprints for KiCad. I’m thinking I would setup a trigger on an INSERT for the parts table, to flag that new components have been added - which after an hour or so cooldown (to make sure i’m doing adding stuff) would get picked up and cause the new library parts to be generated and pushed to github automatically. This way, the KiCad side of things would be fully automated.

There’s already code in place to keep the webserver’s repo synced to github (since I use a basic decoding of the altium files to give me dropdown selections of which symbols and footprints to map). So the web server has access to the library files for that side of things.

It sounds as though generating the KiCad parts wont be too hard, but I’d need to figure out how to manage them - both qty wise, and naming convention wise. Does KiCad have any sort of filtering capability in it for searching for parts?

1 Like
10

more mod comment:
I had to allow the one-line post from @issus with the forgotten link, as the system had flagged it.
But didn’t check if the system also had the whole thread unlisted. I could see it on a main page, but no idea what it’s status was then (~12 hours ago).

11

This forum is the official unofficial KiCad forum for users (yes, it makes sense).

@Rene_Poschl does quite a bit of official KiCad library work.

@issus it appears that a user/system flagged your OP for some reason (glad it wasn’t me). You should get much quicker responses to your queries now that your thread seems to be back to normal.

An open source database library for KiCad does not have to be 100% KiCad KLC compliant. KiCad is still growing and changing at an even faster pace. You may even have ideas to help give direction as it continues to improve. The official place to share your ideas is the Developers Mailing List; I’m certain you can find the link on your own.

Thanks for joining the forum!

12

I do not official work here on the forum. (Other maintainers need to have the option to give input. This is only the case if the issue is handled over on github.)

Naming convention is always hard to decide (i know because i wrote huge parts of the official library convention)
You could use the KLC or industry standards as a starting point (To avoid starting from scratch and making the same mistakes as others did before you)

One thing you will need to decide early on is if you want the namings to be easily human readable or if you want to have short names (KLC goes towards human readable, IPC towards short names)

Only on the name and tag fields and only in text form. (Text form means it can use wildcards similar to how the linux filename substitution works. * for any number of chars, ? for exactly one char.)
So you will need to come up with a well documented naming convention for users to be able to easily search for parts.

13

And even more comments regarding mod stuff:

@ChrisGammell took care of unhiding the initial post. He answered to the message that i sent to the moderators group with details. I would have assumed that you guys also get such a message. I think in the future i will manually create a message with all moderators as the group messaging seems to be kind of screwed up.

14

Hmm, I suspect that perhaps KiCad might not be ready for such a library then (and I say this in the nicest way possible!). The huge advantage in Altium with the dblib (other than placing real parts tied to a supplier) is the ability to filter by parametric values. Like my capacitor example - go to the Capacitors MLCC view, filter by 0603, filter by 0.1uf, filter by 16v capacitors, sort by price and choose the cheapest.

Without filtering, and having over 5000 resistors in the library alone (01005 to 1206, with every value in 1%, 0.5%, 0.25%, 0.1% and 0.05% from several suppliers) it would make it very very difficult to find the part you need.

So, what can we do to get this functionality into KiCad?

  • Does KiCad support plugins, so we could develop a KiCad plugin for the database library?
  • If built in functionality was developed by the community (assuming someone is willing to help) would this be integerated into the main release, despite not being on the roadmap?
  • Would there be enough interest in the community in the library to get help with integrating it? I’ve had a lot of messages from people saying they would love to see the functionality in KiCad, but that does equate to help in implementing it.

I’m happy to help out as I can, but my software dev experience is all in .net on Windows, and C on microcontrollers. I don’t feel I have the experience to implement this myself. I feel I should be able to tackle the footprint and symbol porting, just not the KiCad modifications.

I’d be happy to port the library over to MySql (basically have a MySql mirror running on our server along side the main MSSQL instance) - OR - there is a JSON web api for the database, which could be extended and used rather than a direct database connection.

15

to be honest i am not really convinced that it is a good idea to have one tool for everything. In my mind part management is best kept separate to the circuit design tool. So this would mean you have a program that has your part management that you use to select your component and then either push the information about the selected component into kicad directly or provide a primary key that is then used for interfacing (To get a BOM out of the design tool.)

This will allow you to have the part management tool specialized for your company internal ordering system (in a lot of cases this will mean interfacing with SAP) and any changes to that system do not require any changes to your CAD program.

Maybe this is just a difference in thinking as i would say the design should drive BOM not the other way round.

16

You’re right in your thinking - and thats how the DBLib works with Altium. There’s no ‘local stock on hand’ capacity in the library - it’s just a huge database of footprints and symbols related to parts. The only reason it would need changes to the CAD package is because it works so differently from an integrated library (to use Altiums terms) which has a symbol associated with a footprint to make a specific part - as this leads to generic components for passives and such.

So in Altium without the DBLib I’d place an 0603 sized resistor and set it’s value to 10k. Then in my BOM i’d just have a line for RES 0603 10k. With the DBLib I go and find a Panasonic 10k 1% 0603 resistor and place that on the board. When my BOM is generated, it has RES 0603 10k 1%, with the manufactuer Panasonic, Part Number pn10k1pc (made up lol), Digikey as the supplier, the DK SKU being P10K-1-ND (again, made up) and a link to the part for ordering.

This doesn’t sound like a huge advantage until you try to place an 0402 Cap that’s 47uF 50V… and find it doesnt exist. So when placing parts I can just rely on that part to exist, not have to stop my ECAD work and go over to a supplier website to look up a potential part, then place it, edit it to set the part number or value or whatever. Furthermore, when you start looking at say a MOSFET which has a generic footprint, you have to go back through during BOM editing and figure out which one is going to be best - OR - do that at the time of placement, again taking you from ECAD over to your browser and you need to edit the value of the part so it relates to the part you just chose from the supplier website.

The DBLib significantly cuts down on mistakes and significantly speeds up design time, for me at least. It means I know I’m building the board I’ve just spent weeks on as it was designed in a schematic, and if I’m not sure of a part when I am assembling the board I can click on the symbol in the PDF of the schematic and it takes me straight to the datasheet for that part. So debugging/diagnostics are faster too. I like that it lets me get on with the busines of designing a schematic or board, rather than faffing around figuring out parts or editing a BOM just so I can order it.

Setting up a BOM for ordering/manufacture went from being a whole days work on a 200-300line BOM to being a 15 minute job (because Altium sucks at accessing remote databases efficiently haha) which doesn’t require any interaction from me.

Attached is a PDF print of an Excel BOM out of Altium for an open source board I made. No editing, no modification required. Without the DBLib, the description column would just be the Value field of the component (ie: 0.1uf) and the manufacturer/pn/supplier/sku/price columns would all be missing and need to be filled out manually.

A very large company might be using SAP, but your average student, hobbyist, small and medium business will not and has no ERP system. So for me, this is a huge time saving on the design side of things because I can place real parts into my schematic, and I can order my BOM really quickly (ie: copy and paste from the spreadsheet into digikey and click order). I would like to eventually build in the functionality to filter to just parts you have in your parts bin/use as standard (so you don’t end up with 6 different reels of 10k resistors haha), but I’m not there yet.

Feeder Board.pdf (105.4 KB)

1 Like
17

This will work the same even if the tool is external. there the workflow would then be to look in your external tool what you want to place, generate the symbol/footprint pairing (a good tool will try to use as many generic assets here as possible to cut down on library maintainance)
and insert that generated pairing into the schematic. Yes there is one more step but it will mean that this whole workflow is a lot more flexible when compared to integrating everything into one single tool.

18

Well, I guess if the interest doesn’t exist I’ll just leave it there then. I’m not a KiCad user myself, so there isn’t much point in me putting a ton of work into it if the solution isn’t going to be easy/smooth for the end user’s experience.

19

Just a heads up: I am not a dev. Ask over on the mailing list or the bugtracker if you need new kicad features to implement your thing. I can only tell you my opinion that might be biased as i am the main library maintainer.

20

Hi @issus, having a db in kicad would be nice…
you may find this thread useful: