Yet another release date 6.x question

Community
21

The old thread is educating and painful reading. I suggest starting right from the RC1 (or almost) to save some time:

Compare that mess with what RC is expected to mean: a candidate for release as is, no bugs which are planned or tried to be fixed before the final release anymore. (KiCad has bad history of adding new bugs with bugfixes.) Even after the 5.0.0 release there were problems with packaging and for a while people didn’t know which version they had, buggy or fixed.

This really should be considered seriously. Beta quality isn’t RC quality and it’s misleading and bad PR to call RC something which isn’t nearly close to a release. Learn from the past, announce beta versions and keep RC for actual release candidates, and play safe so that even packaging is done and tested before the last RC so that only the version string can be changed for the final release (unless new serious bugs are found, of course). All minor bugs should be postponed to 6.0.1 even just before the RC1 and between each RC. Then the 6.0.1 released soon after 6.0.0.

Otherwise we will see another several months cycle between RC and final. I would rather see long beta phase and short RC cycle. That’s how the rest of the world uses the words.

4 Likes
22

I couldn’t agree more… but I guess it’s up to the developers, not the users and/or testers.

23

5.99 is still in development phase, where new implementations are worked upon. To me, that’s an “Alpha” stage.
If everything planned will be implemented, we’ll go to the “Beta” stage which is “feature freeze”. In this stage no features are implemented, and all efforts go to bug fixing.
RC stage to me means, no KNOWN bugs targeted for release are listed (so if there are any bugs left, they’re minor and scheduled for next releases to avoid regressions). RC stage should have as much workload as possible. If any showstopper is found, it will be fixed before release. Otherwise, green light for the release.
In this scenario, RC stage can be rather short as we’re not fixing bugs nor issue new bugreports; we should just confirm the readiness of RC for release. Anything minor goes to next releases, anything major (hopefully not) takes us back on schedule.

EDIT: seems that I’m moreorless right with my assumptions on Alpha/Beta/RC:

24

Tbh, I don’t think changing the names or meanings will change the process much. KiCad developers will continue to fix bugs, and tag releases when they think the code is ready.

Otoh, ‘adventurous’ users will continue to use nightly builds, regardless of release cycle. ‘Conservative’ users will only use builds tagged final release. ‘Helpful’ users will report bugs when they encounter them.

Therefore, declaring builds as “beta” or not is unlikely to significantly change how people use KiCad, nor make it more or less likely that users report bugs.

Unless KiCad starts to include some telemetry, developers don’t have a good way of telling what test coverage has been achieved, so it’s pretty much gut feeling.

8 Likes
25

You may be right. But still using more realistic naming together with better infrastructure (gitlab allows communication, milestones etc. better than launchpad), grown userbase, donations, the new company offering commercial support and bugfixes, more disciplined and strict planning etc. may help people orient better and have stronger trust in the project development cycle than the last time. And that would draw in more testers when new people get interested.

Even a couple of new active testers for beta phase (and of course the RC phase) can make a difference. 20 new active testers would be groundbreaking.

In the v5.0rc cycle the amount of commits which marked some bug in the database fixed was something like 250…300 between RC1 and RC2. Not counting other commits which weren’t marked but which fixed or changed something found by a developer. Anyone can understand that this is not a “release candidate”. I have already ranted too much and I want everybody know I really appreciate and admire the work the developers have done. But still I must say that this is just bad planning.

I know very well that the old cliche “to be taken seriously in professional world KiCad must…” is what it is, but if the project want’s to be professional, the release cycle must be professional, too. Whether or not it affects the actual outcome much. Version strings don’t make that difference alone but are still part of it because they are part of the planning and executing the cycle.

For the last RCs, on the other hand, the testing and new reports by users were much more important. If there comes a new crash report every now and then, say once a week, and the previous one is difficult and takes one week to fix, ten reports is enough to postpone the release two months. That’s not the fault of the testers, either, but it reveals lack of testing. Most of those bugs should be known in the beta phase.

Even when changing the names or meanings may not change the process much, because they are easy to change, why not do it? What happens between tags/names is of course more important and more difficult provided that the changes to the codebase are actually planned beforehand, and planning is what really counts. I have already seen it getting better for gitlab milestones compared to 5.0 and launchpad times.

The rest, testing and reporting, is up to us all. Ideas for making that better are welcome. Maybe we should start a new thread for that (this one has been hijacked for too long already).

3 Likes
26

What I said keeps haunting me. I have to apologize, I’m sorry.

The decisions back then were made using the available knowledge. The situation was unique. It wasn’t “bad planning” then, and that’s not what I meant. Although I obviously think it didn’t go perfectly, but everyone agrees with that.

We have more data and facts now to predict better what may happen. If and only if RC means what it normally means (ready to be released unless new serious bugs are found) to all parties involved and the final is released after six months from the first RC and after hundreds of changes, that would be bad planning now. We can learn from history. I believe it goes differently this time.

If the project wants to keep the aberrant meaning of RC, then OK. I just think that names are not only technical tags in the database. They are communication. Communication is based on common understanding of words. The project is growing bigger and more professional and needs communication which is more and more professional. Therefore I wish that the commonly accepted meanings of terms are accepted in the project even if it itself isn’t critical for the release schedule.

1 Like
27

People keep saying “more professional” but many professional software packages have loads of bugs.

There are always two things to deal with

  1. what people say they want
  2. what they actually need

People want a warm fuzzy feeling about the product, so marketing buzzwords can do that. But what they really need is bug free code and improved features. Marketing buzzwords don’t help with that.

There are several key things about software, which have remained true in the decades I have been in the industry:

  1. catch 22: You don’t want to release code to a wider audience until the bugs are found, but you can’t find bugs until you release the code to a wider audience.
  2. Writing software is really about learning. i.e humans slowly learning what works and what doesn’t. There is no pill to take which gives instant knowledge.
  3. Plans nearly always fail, because we don’t know what we don’t know. Unless you are doing the exact same thing you did before, you can’t be sure. By definition, if you are creating new things, there will be new unknowns.
  4. Often the best way to help developers is just to let them get on with it.
2 Likes
28

This is generally not true. You can catch vast majority of bugs using rigorous automatic testing. But this requires developer resources, machine resources and a polished dev process. Kicad like most open source projects unfortunately doesn’t have any of those.

29

Generally, projects do not have rigorous automatic testing. So yes it is generally true.

QED

I guess there is also
5) despite pointing out obvious realities, people immediately deny them. The first step to solving a problem is recognising the problem.

1 Like
30

It’s not true because it’s not a catch 22 when there is a 3rd valid option available: build rigorous automatic tests. I agree with the general message, specifically if applied to kicad. Still wouldn’t generalize it to all software though.

31

I think you are nitpicking, people seem to love arguing over semantics. My point of “a wider audience” was “to get wider test coverage”. That can be achieved by manual test or automated. An intelligent developer would understand that, and I think you to do.

In an ideal world, automation tests everything. Of course, there is still the problem that you don’t know everything that needs testing. Back in the real world, none of the professional projects I have worked on have had rigorous automatic testing. But we can always dream.

The fact that in real world projects you don’t know what bugs there until you test remains true, which was the point. I am sure you will find a way to disagree with that too.

1 Like
32

Ok, now you are just twisting the meaning of your own words which no reasonable person would understand the way you are suggesting, take a look at what I initially quoted few posts above.

Back in the real world I’ve had 3 jobs in my career so far, first company was not a software company, just had a few in-house devs, there was pretty much no automated testing. The other two were built by software engineers and every single project was rigorously covered by multiple layers of automatic testing, it just wouldn’t get released otherwise. So it depends.

Oh I agree. But like I said most bugs can be caught with tests without any end user ever seeing the software. And it’s usually those bugs that are easily caught that are most glaring and make an impression of unprofessional software when users experience them.

33

Come on, you know the point of beta releases is to get wider test coverage. Even non-developers know that.

These side discussions get very pedantic, and non-constructive. It seem to be more about proving who is technically correct, than achieving a common understanding.

34

Yeah and you just keep ignoring the fact that I only responded to one specific point of yours and don’t disagree with you in general.
Anyway, it does get silly and derails the topic. Subject closed.

35

You mean that you guys get to release bug free software!? Nice!

36

It’s not easy to avoid all bugs, even with automated testing. But with a good test suite it’s definitely possible to avoid regressions. You shouldn’t need to ever “retest” a new release, once you encounter a bug, write a test case and it’s caught forever.

also, I personally was happy using nightlies until I heard that there was a possibility the new file format would have breaking changes that could be non-forward (not backward!) compatible. I think a commitment from the developers to avoid such a situation, (except perhaps in specific feature development branches) would help get more eyes on nightlies.

1 Like
37

That’s new to me. Do you mean that file created by current Nightlies would be un-openable by later releases? Any sources? AFAIK there’s an unimplemented feature pending.

38

There’s no promise it won’t happen. IIRC it has happened maybe once. I vaguely remember one discussion where this was at least a possible reason for a user’s problem.

I think this is one good reason to announce a beta phase soon after the file format is stabilized and the last major features are roughly implemented, but before the rc phase. It would be a clear signal for those who need relative safety and combatibility but can stand bugs and occasional crashes. I’d guess it’s the majority of potential testers, at least if we think of more than occasional isolated new feature tryouts.

39

According to @craftyjon, the situation that the files could not be opened by future versions is “extremely unlikely”. So fingers crossed :wink:
But indeed, unless Beta testing will be announced (with true “feature freeze”), 5.99 is not for the faint-hearted.
“Beware, dragons ahead”.

40

I quote Jon from another thread because the discussion was actually started here and doesn’t belong to that other thread:

Thanks for the feedback. I understand what you are saying. I’m very well aware of the catch 22. That’s why I have encouraged people to test and report early. However, I don’t see things being having been changes so much that it wouldn’t be easy to predict the same happens again. Therefore using ‘RC’ in the similar situation would just lead to a similar RC cycle again.

Announcing a beta or several betas might help a bit so that naming would be realistic, yet it could attract testers. Soon after the first big features have been merged (at least curved polygons which affect the file format) would be a good time to announce the first beta, which would also signal the end of the planned file format changes. If the version string is changed to for example 5.99.1-beta all nightly builds with that version could easily be identified as file format complete. Then, if ff-exceptions are still merged later, 5.99.2-beta would be announced. Especially python API freeze, if that comes, would be a good reason for a new beta. When there’s temptation to announce RC1 it should be resisted and the final beta announced – just because we can know for sure that new bugs will be found. Then the RC1 would be announced with a hope that no 5 RC’s and several months wouldn’t be needed anymore.

This way the version naming wouldn’t be arbitrary, it would actually communicate something meaningful in the development process, and at the same time might attract more testers. As we know, names and versions have psychological effects; meaningful versioning would give people something to grasp.