Contribution to symbol and/or footprint libraries still welcome?

I love this comment! Or I did the first 273 times I heard it…

Since my point wasn’t clear, I’ll restate it: If you volunteer to manage the libraries, then manage the libraries. If you can’t, then just say so and release control to someone else. There’s no shame in that.

I’ve been in that situation. I created KiCost and then supported it for a few years until I just got tired of it. Luckily, @hildogjr wanted to take over and he’s done a great job supporting and extending it. I think that was a better solution than just ignoring issues and pull requests while the tool rotted. The same is true for the libraries.

And ghosting the libraries has a real cost. It’s a pain in the ass to follow the KLC only to have your submission ignored. That makes a person who put in the effort to support KiCad just a bit less positive towards it. And since KiCad doesn’t have a multimillion-dollar marketing budget, it needs those positive feelings so members of its community will recommend it to others. That’s less likely if they’re pissed off.

telling volunteers to volunteer more of their time isn’t likely to win you much sympathy.

Valid point, but isn’t it a numbers game on the librarians ?

Could the waiting be put into a ‘pending’ library with a proper disclaimer?

IDK, maybe. The point is that the current system isn’t working well. Instead of cranking away and falling further behind, new solutions are needed:

  1. Include “pre-release” libraries in the KiCad distribution.
  2. Create a search capability to find and load parts stored in repositories on Github/Gitlab.
  3. Create an enhanced version of KiPart that supports submissions in a simplified format and generates KLC-compliant symbols automatically.
  4. Or just about anything else…
2 Likes

And who is going to read the disclaimer?

I can envisage a forum full of:

I used this symbol/footprint from the Kicad library and it’s broken.
I thought Kicad libraries were supposed to work.

@devbisme
Unfortunately I’d have to include your ideas with Hermits, for the same reason.

I really have trouble understanding the reasons for contributors complaints.
They create a symbol/footprint, they place it in their library, it is theirs to use forever: Great!
They wish to let others use it so: Greater! It will happen, in time. Why all the fuss and dummy-spitting?

I don’t continually read:
I had a great idea for Kicad so I made a feature request and it hasn’t been implemented in twelve months?
OR:
Why isn’t the documentation complete and up to date?

Why all the carry-on with libraries?

I think because of the effort required to meet KLC and then the complete lack of acknowledgement/feedback. (I’m guessing since i’ve never submitted to the libraries and am just going from replies to this thread.)

1 Like

Meeting KLC makes nicer and cleaner looking schematics and PCB designs. And it does not take to much effort.

Asking why people complain is pointless. This would be the same as asking why to contribute to an open source project at all!

KiCad and also its libraries have been (and still are?) an open source project relying on contributions. More on the code side; less on libraries but still both are needed.

And now we have two extremes:

  1. Too much contribution (which no one can coordinate)
  2. No contribution at all

Obviously both extremes do not work well.

Having noticed that the number of MRs for the libraries is increasing constantly
I “just” asked if contributions are still welcome (See #1). Nothing more, nothing less;
no complaint at this time! However this turned into a vivid discussion leading to the
conclusion that the current system is broken. And maybe this discussion leads
to a future change that improves the workflow/process - for maintainers and contributors!?

Getting back to a reason for complaining:

Most of the parts I use are not contained in any official library. And I don’t care too much as suggested.
But if I use official symbols or footprints which need fixes or extensions (e.g. derivations, KLC errors) I always need to be careful to not use the broken one from the official lib - and everytime I install a new version the trouble starts again. Why not fix the broken part instead of creating a fixed clone
and avoid the trouble at all?

Sometimes if I think that a symbol might be useful for others or used regularly I contemplate to contribute it…

I was curious about how much is going on in the libraries, so I had a look at the link below, and I count 170 posts for this month (now 17 days, so 10 posts per day on average), and this is only for the footprints.

Moreover, looking at the repository, it looks like they are doing a pretty good job.

Footprint

Symbols

Whats more, you’ll only see all this changes if you install the libraries with each release or pull the repository down by yourself.

I agree that the KLC serves a valuable purpose in that it prevents the libraries from becoming wildly inconsistent. And maybe I’m wrong that they are onerous. (They certainly aren’t as bad as stuff like IPC standards.)

2 Likes

I apologize to the maintainers for saying they were not adequately managing the libraries. I was wrong.

Whatever the reason for the backlog in merge requests, it’s more complex than I thought.

2 Likes

If I may make a comment as (newish) librarian: a review of an MR takes about 10-20 minutes if there are no comments to make and the contributor provided all of these:

  • screenshots of the footprint or symbol:
    • footprints: fully dimensioned drawing and a 3D fit-up with the manufacturer 3D model if available
    • symbols: screenshot of the part in a representative schematic unless trivial
  • datasheet link
  • screenshots of relevant datasheet extracts
  • explanation of design decisions that you made
  • reasons for KLC exceptions (or a green pipeline tick).

If there any comments, any questions, an unexplained KLC pipeline failure, or any uncertainty, then typing out the comments, creating screenshots, that MR will now take at least an hour of keyboard time, but also likely means a round-trip which will delay the whole process by days. In the gap between arriving at home after work and going to bed, even on a day when I can carve out time for reviews on top of other KiCad-ery, let alone my own projects, I can rarely hope to complete a single review.

On Sunday, I spent basically a whole day and started 8 or 9 reviews. None of them could be merged that day. Only one of them could be merged after the first review, all the others needed another go around. Out of the 507 open footprint MRs, I currently know of only two, maybe three, that probably can merge without further comment. Two of those are my reviews from Sunday that I need to double check before pressing the button.

The “one simple trick” you can do to getting something actually reviewed is to convince a reviewer stepping out of the car after 10 hours in the office that your MR can be merged before bedtime. You can do this by adding all the items above, which indicate firstly that you’re willing to make the effort to make the MR easy to review and second that you have actually checked the footprint or symbol is correct and complies with the KLC. Pre-empt every question a reviewer could have. Remember, the reviewer is expected to stand behind every single part they review as if they drew it themselves.

Aim for an MR that needs no comments. These do happen! Honest! Personally, I love it! I don’t like nitpicking MRs, it’s boring and I feel bad for asking for changes! Look here (for footprints): Merge requests · KiCad / KiCad Libraries / KiCad Footprints · GitLab - any MR with only one comment merged without further changes (the one comment would be “merged! thanks!”).

15 Likes

A small remark on this: looking through the recent merge requests it seems a bit like there are no guidelines on gitlab directly on how the ideal merge request would look like? Gitlab has this great feature where you can set templates for merge requests and I think this would help both sides here greatly. The ones requesting could then easily see what information they have to provide and the reviewers can check if the required information was given and reject all others on this base.

1 Like

They are templates. However they only appear when you open a MR. It’s basic stuff like a few screen shots and datasheet link. Easy to gather.

John, thanks for the details on what it takes to merge to the library.

A few years ago, I built several symbol libraries for multiple FPGA families across multiple companies. There were hundreds of parts, some having thousands of pins. These were all done with KiPart being driven from pin tables directly from the manufacturers.

If I had submitted these as libraries (I did not), would they ever have been merged? Manually verifying the pins would have taken months! Nobody would want to do that if there were easier parts to merge. Is the backlog caused by passing over more complicated merges, or is there some other factor that’s the main cause?

The disclaimer is in the library name. Not a ‘solid’ suggestion to start discourse.

Hi!

In the specific case of many complicated FPGA parts, the best way would likely be to make a generator. We don’t have many of those for symbols, but there are a few. Notably, the large libraries of STM32s are indeed generated, and were merged (in fact any footprint or symbol added in the last many year will have been reviewed and then merged - nothing skips review, no matter who made it). Probably the best way to have approached that would be to submit a handful at first and make sure the general pattern is right in some limited, reviewable-by-mortals sense, then expand to the whole range.

The backlog in any queue is always simply caused by a fundamental result of queueing theory - as the average load factor of a queue’s service agents (i.e. in this case how much of the total review capacity we are using) approaches 1, the latency through the queue (i.e. time from MR submission to merge) tends to infinity. Of course, once a queue is “in the hole”, the load average is pegged to 1.0 while the backlog is handled and latency increases without bound until you either clear the queue somehow or die trying (We are here!)

There can only ever be only three ways to resolve this: increase service agents (add reviewer-hours), reduce requests (e.g. refuse some MRs entirely - not ideal!) or reduce load per request (make MRs quicker to review).

Communicating with contributors in a “nice” way (rather than being terse and dismissive of the effort they made to submit the MR) takes, literally, hours and hours. In order of total time spent: my feeling of where the time usually goes:

  • typing the comments, taking screenshots and double checking everything so I don’t give misleading, vague or confusing comments and frustrate the contributor too much
  • digging up details e.g. signing up for websites to get 3D models to do my own fit-ups, finding applicable standards like JEDEC ones, opening models in FreeCAD and measuring them, finding application notes to get an idea of what is correct part usage, that kind of thing
  • actually checking the “obvious” symbol//footprint content by making my own dimensioned drawings or reading the datasheet pinouts
  • switching back and forth between different MR branches and keeping track of returning review comments - every time I do this, it’s a context switch
  • checking with other librarians when I’m not sure in edge cases

Anything an MR submitter can do to reduce any of those helps enormously.

It’s likely that the remaining MRs do skew to the more complex, because easy ones often get picked off quickly what a reviewer has 10 minutes to spare and doesn’t want to get involved in a huge review. That said there are quite a few simple ones there that often just look “unappealing” (relative to others, or doing ones own work). Footprints and symbols are very complicated and even the simplest have a large number of things that are often not quite right. The review comments for even the simplest MR needing revisions can contain 5-10 points. A complex review that takes 2 hours to verify but is actually right in the first instance is always easier than a trivial-seeming MR that still somehow requires a two-week-long back-and-forth process to slowly converge on a mergeable result like I’m playing an excruciating game of 20-Questions by post.

A surefire way to make your MR suspected to be of the second type is if the KLC check is not green and just a perfunctory message like this:

Add ABCV4533E
url: https://someco.com/ds/sdfsdfsdfsdf.pdf

Remember, every MR that takes 1 hour (already optimistic for an MR that isn’t one-and-done) blocks 6 requests that each take 10 minutes. Even if we say only 3, look at the difference on that graph if we could have 0.33 load factor rather than 1.0.

Now, as for what you can do: any MR you make - make them a 10 minute one. Check them twice, then check them again. Provide all the materials to show your working. Guess what a reviewer could conceivably ask and answer it first. If the KLC checks fail, do not wait for a reviewer to ask you to deal with it. If the manufacturer 3D model doesn’t seem to fit, fix it first. If the dimensioned drawing doesn’t have the same exact dimensions as the datasheet, fix it now. If GitLab tells your the rebase won’t work - now is the time to deal with it. If you know the MR is not ready and you can’t fix it now, you can mark it as a draft in the meantime.

If you want to help clear the queue, you can do a review of another’s MR on your own, even as a non-librarian (and you can become a librarian this way…I should know!). It’s not rude, it’s very helpful. You can add missing dimensioned drawings, 3D fit-ups and check the KLC pipeline results and other manual checks. If we can clear down the queue, the load factor won’t be 1 any more because we won’t be filling every spare moment with the servicing the backlog and the latency will be far, far lower. The other thing that you personally will learn in that process is what a reviewer is looking for, and then you will be able to pull off the sacred and mystic art of the bullseye MR when it comes to your own submissions!

Another thing you can do is to make (simple, clear, easily-reviewed - see all of above!) MRs to make the library more consistent and KLC-y a small chunk at a time (here’s one I did, for example: Rework Lumberg 1503-07 THT audio jack (!3281) · Merge requests · KiCad / KiCad Libraries / KiCad Footprints · GitLab). This helps because when others, quite reasonably, use existing parts as a basis for new ones, we won’t have to spend scarce review time correcting the KLC issues they inherited from an old library part that predates the KLC rules in question.

Disclaimer: this is all my own opinion - librarians are their own people and I certainly don’t claim to speak for them or for the library team as a whole.

9 Likes

Hi John.

As I am sure would many, I would like to be able to help in any way I could.
But the entire process seems daunting.

Would any of the existing reviewers consider mentoring?

Take care.

M

2 Likes

Hello @VillageIdiot , thank you for:

I would like to be able to help in any way I could

Please check your messages.

Best regards,
Aristeidis