New project file format

[craftyjon]

Hello everyone!

A new project file format has been merged and will show up in nightly builds soon.

About the New Format

The new project file format is JSON-based and has the kicad_pro extension. Switching to JSON makes it easier to store structured data, and changing the file extension makes it easier to tell the two formats apart and also fixes the issue that the old .pro extension is used by a lot of different applications.

In addition to changing what the file looks like, we have also changed what information is stored in the project. Some things that were not really useful anymore were removed, and some information that was previously stored in the board file was moved to the project file (including design rules, net class information, and editor defaults).

Project local settings files

You will also notice a second file created with the .kicad_prl extension. This is called the Project Local Settings file, and will store settings that should be considered “local” to your computer. For example, which layers are visible in the board file.

These settings are put in a separate file because they can change often as you use KiCad, and two users working with the same project may want to have a different view of it. If you use a version control system like Git to track your projects, the .kicad_prl file is the kind of thing you may want to add to your ignored files list so that it does not get checked in.

The number of things stored in this file will grow over time, so that KiCad is better able to save and restore your session.

Notes to nightly build users

When you update to the latest nightly and open a project, it will be automatically converted to the new format. The old project file (.pro file) will be left alone. If you open a board file, the design rule information in that board will be moved to the associated project file and removed from the board. This means that, like with all board file format changes, you will not be able to open that board anymore with older versions of KiCad.

While most things should be migrated, there are a few exceptions: if you have made use of Jeff’s recent additions of DRC/ERC severity settings, you’ll need to set those again.

As always, we encourage nightly build users to test KiCad with copies of your projects and/or use a version control system to store your projects, so that you can always revert back to an older version if you need to.

Thank you for testing KiCad, and as always, please report any bugs you find with this or any other new feature over at the GitLab issue tracker.

A note about backup files

One other notable change is that the backup file system has been removed. This was the system that would create kicad_sch-bak and kicad_pcb-bak files every time you save. The short story about why this is removed is that with recent changes to the way file saving works, it should no longer be possible for files to be corrupted if KiCad crashes during a save, and the generation of these backup files was seen by many users as annoying clutter. For more context about this decision, you can read the thread on the developers mailing list.

An new backup system that properly backs up the whole project (see the GitLab issue) has been implemented to replace this feature. As always when using nightly builds, back up your files separately in case a KiCad bug breaks the built-in backup system.

Cheers,
Jon

[maui]

@craftyjon

One other notable change is that the backup file system has been removed.

@craftyjon

The short story about why this is removed is that with recent changes to the way file saving works, it should no longer be possible for files to be corrupted if KiCad crashes during a save,

@craftyjon

We do have a wishlist item to create a “real” project backup system (one that backs up all the files in a project to an archive at a certain frequency, etc). Until KiCad supports that, we encourage users to make use of some kind of backup strategy such as a VCS or a cloud backup tool.

Removing a backup strategy (even if minimal) before adding a new one is IMO not a useful move.
Quoting JP here:

Backups, like any security system, bothers you as long as you do not need to use them.
But you are happy to find them in case of trouble.
…
For me, backups are like a accident insurance: you need them and you hope never use them.
Just my two cents.

[craftyjon]

Normally I would agree with you, however in this case I think the old backup system needs to be removed until another one is put into place because the old system had flaws that make it a really poor backup. I would rather people not think that they can rely on it only to learn of these flaws.

It seems like there is enough interest in a real backup system that it makes sense to include in V6 (it should not be that big a job to put in place a basic system that actually backs up everything together), so this will only affect nightlies for a period.

[maui]

@craftyjon

Normally I would agree with you

good

@craftyjon

however

there is always a ‘however’

@craftyjon

in this case I think the old backup system needs to be removed until another one is put into place because the old system had flaws that make it a really poor backup. I would rather people not think that they can rely on it only to learn of these flaws.

as I said a backup (even if poor) it is better then nothing.

@craftyjon

It seems like there is enough interest in a real backup system that it makes sense to include in V6 (it should not be that big a job to put in place a basic system that actually backs up everything together), so this will only affect nightlies for a period.

I wouldn’t have removed the (poor) backup mechanism until the new one is ready, even if in a beta release.
Instead I would have opened an issue stating the actual backup procedure has to be changed/improved.
This is the way it should be always be done in creating a new/better feature IMO.

It would be better to re-enable the old behavior until the new one is ready to be merged.

[der.ule]

Looking around in the forum, more than a couple of people have been saved by the current backup system, specially, because they didn’t even know there was an automatic backup, so they could not as easily shoot themselves in the foot. I am all for better systems and improvements, but at the moment, this change removes the (flimsy) safety net currently in place and replaces it with nothing.

[craftyjon]

I hear you. I will put a working backup system in place soon. I think nightly users have to accept this kind of responsibility to keep their own files safe, though, because a nightly could have a bug that corrupts their files anyway (old backup system or no).

[maui]

@craftyjon

I will put a working backup system in place soon.

thanks

[set]

Moving to JSON is very positive. I wonder why not YAML, is much more human readable and support for it is really common.

Having this format, and having not one, but two files: Why we still have two other files to configure the project? I’m talking about fp-lib-table and symbol-lib-table

What’s even worst: Why two different formats? Here I mean: why JSON and S-Expressions? IMHO JSON and YAML have much more support than S-Expressions (BTW from where comes this lisp style format)

I think library mapping should be part of the project, in fact part of the kicad_pro.

BTW I see really ridiculous that fp-lib-table and sym-lib-table contains absolute paths. I think KiCad should automatically use ${KIPRJMOD}, ${KISYSMOD} and ${KICAD_SYMBOL_DIR}.

In general is a very bad idea to use absolute paths:

1. They expose personal information
2. They are useless when sharing the work (using version control or just sending a ZIP to somebody)

[Rene_Poschl]

@set

I think KiCad should automatically use ${KIPRJMOD} , ${KISYSMOD} and ${KICAD_SYMBOL_DIR} .

It does just add a library that is inside any directory that is covered via a path variable by use of the library managers and see what happens.

[craftyjon]

@set

I wonder why not YAML, is much more human readable and support for it is really common.

Because we chose JSON

Here I mean: why JSON and S-Expressions?

Because we are not going to change the design file formats away from S-Expressions just to match the config file formats. All pain, no gain.

Why we still have two other files to configure the project? I’m talking about fp-lib-table and symbol-lib-table

Again, changing this would be all pain, no gain

[JeffYoung]

@set

(BTW from where comes this lisp style format)

Well, from Lisp…

[qu1ck]

@craftyjon

no gain

Arguable. Is it worth the pain? Probably not unless the goal is to get rid of all s-expressions which I would support.

@JeffYoung

Well, from Lisp…

All 3.5 people still using Lisp light up for a moment and then immediately go back to counting brackets

[craftyjon]

“uses a different format that some people have different opinions about” is not a gain

There were technical issues with the old project format, so we changed it. There are not really technical issues with the sexpr formats, just opinions.

[qu1ck]

Well, everything starts with an opinion, just some opinions are backed with stronger arguments than others. I don’t want to go into details here about why I think s-expressions is a poor choice today, I discussed it on the mailing list when I broached the protobufs backed file formats. Most of those arguments were not refuted.
https://lists.launchpad.net/kicad-developers/msg38772.html if someone is interested. Note, I don’t even argue for specific representation there (json vs yaml vs proto text native), just about benefits of using formal interface definition language with vast code gen support.

[craftyjon]

The arguments weren’t refuted because they are valid arguments when considering what format to choose for a new type of file. The problem is that they are technical arguments and ignore the cost to users of the change, and the time cost to the development eam.

The main reason we don’t consider moving the board format, etc from sexpr to something else is that there isn’t some big win for the users that would make it worth the pain of disrupting the whole ecosystem. We knew there would be pain enough moving the schematic format.

The main reason we moved the schematic format to sexpr, rather than moving both it and the board format to some third new system, is because the existing sexpr system is by this point very “battle-tested”. Moving to a new format would introduce a ton of development cost, and a ton of user pain (beyond the existing pain of migrating the symbol libraries and schematics).

That kind of tradeoff doesn’t make sense for a development team the size of KiCad’s. We need to focus on fixing issues that prevent people from doing certain types of designs in KiCad, so that KiCad becomes more widely applicable to people wanting to make PCBs. That means we need to be very careful about considering big changes that don’t advance that goal, and in the case of the design file formats, there just isn’t enough “wrong” with sexprs to make that work out in our favor.

[qu1ck]

I understand and agree on most everything except development costs. KiCad team spends a lot more time on maintaining format parsers and related build issues than it would take to maintain protos. And users spend a lot more time figuring out file formats because there is no central up to date documentation.
I doubt many devs looked at the example patch I provided to see how easy it is to work with protos.

@craftyjon

The main reason we don’t consider moving the board format, etc from sexpr to something else is that there isn’t some big win for the users that would make it worth the pain of disrupting the whole ecosystem. We knew there would be pain enough moving the schematic format.

That last point is exactly why I wanted schematic format to be the first to move away from s-expressions because that pain of switching formats was going to be there anyway.

I totally understand that there is bigger fish to fry and not enough stoves. It’s just a shame that a good opportunity to reduce developer costs in the long time and improve KiCads interoperability with other tools and languages as well as reduce user pain was wasted.

In the future there will be even more friction and inertia to overcome as KiCad grows, but hopefully some day.

[craftyjon]

@qu1ck

KiCad team spends a lot more time on maintaining format parsers and related build issues than it would take to maintain protos

Do you have evidence for that? It doesn’t really feel like that from inside the team.

I doubt many devs looked at the example patch I provided to see how easy it is to work with protos.

I have worked with protos. It’s not about how easy it would be, it’s about how hard it would be to move away from what we have today to something (anything) else, and whether or not there is sufficient benefit to make it worth it.

And users spend a lot more time figuring out file formats because there is no central up to date documentation.

Making it easy for users to figure out file formats is not our top priority. Users should use the program, the program should read and write files. Users reading and writing files isn’t a high priority.

I do admit that the file format documentation could be better, to aid those writing programs that have to parse KiCad files externally. We’ll do another improvement pass at this after feature freeze.

That last point is exactly why I wanted schematic format to be the first to move away from s-expressions because that pain of switching formats was going to be there anyway.

But then we would have to integrate some entirely new format system, instead of just re-using the existing sexpr system, and creating the new format would have taken 2-3x longer at minimum.

It’s just a shame that a good opportunity to reduce developer costs in the long time and improve KiCads interoperability with other tools and languages as well as reduce user pain was wasted.

Again, from inside the dev team, the current system doesn’t seem nearly as high-cost as you make it out to be.

[qu1ck]

@craftyjon

Do you have evidence for that?

The fact that every new field requires N lines of code to be written and tested is already a lot more effort. Project file format already has schema migration code (and bugs) that wouldn’t be needed with protos.
Re: build issues, I’m regularly getting my source tree modified by build and it’s usually lexer files.

@craftyjon

It’s not about how easy it would be, it’s about how hard it would be to move away from what we have today to something (anything) else

Yes, there are switching costs. That’s why I’m saying that development effort benefit is in the long term, not immediate. And for schematic format you have already committed to investing that effort anyway.

@craftyjon

Making it easy for users to figure out file formats is not our top priority. Users should use the program, the program should read and write files. Users reading and writing files isn’t a high priority.

Yes, it’s not the top priority but why not choose a format that gives you this for free? And it’s not so much about humans reading/writing the files but about other tools doing it.

@craftyjon

I do admit that the file format documentation could be better, to aid those writing programs that have to parse KiCad files externally. We’ll do another improvement pass at this after feature freeze.

I trust you will. But then it’s only a matter of time when the doc will be out of date again. And will likely not maintain differences between v6 and v6.99 and later versions.

@craftyjon

But then we would have to integrate some entirely new format system, instead of just re-using the existing sexpr system, and creating the new format would have taken 2-3x longer at minimum.

@craftyjon

Again, from inside the dev team, the current system doesn’t seem nearly as high-cost as you make it out to be.

I already acknowledged that there is up front cost with regards to a switch. I think code generated parsers alone is enough of a benefit but if you say it’s not that bad I will not argue.

But the thing is the more you will use protobufs the more you will gain.
I’ve read you plan to replace the KiMail contraption with rpc based protocol? Protobufs were made for this.
You want stable python API divorced from c++ internals? Protobufs were made for this.
You want a database library interface? You guessed it, protobufs will take care of packing your data.

Can you do it without protobufs? Of course, but you can’t argue that they don’t reduce development costs overall.

[craftyjon]

The fact that every new field requires N lines of code to be written and tested is already a lot more effort. Project file format already has schema migration code (and bugs) that wouldn’t be needed with protos.
Re: build issues, I’m regularly getting my source tree modified by build and it’s usually lexer files.

I don’t dispute that we write more code than we would with some format that involved automatic code generation. I do dispute that we spend a lot of time fixing bugs with the sexpr parser. (also the “modified build tree” thing seems like an issue with your local setup somehow – that is not normal)

And will likely not maintain differences between v6 and v6.99 and later versions.

It is true, we will likely not maintain good format documentation during any development cycle (or at least, not with our current amount of developer time available), only for stable releases. We also want the freedom to make breaking changes to formats during development cycles, so it seems like there is little point to writing good documentation because it will give people depending on the file formats for external tools a false sense of stability.

@qu1ck

I’ve read you plan to replace the KiMail contraption with rpc based protocol? Protobufs were made for this.

Nobody has started on this and it’s an area where experimentation is lower-cost. I would suggest if you really want to push the idea of protos forward, this is where you start. We’d be happy to have your help with this if you’re up for it.

[qu1ck]

@craftyjon

It is true, we will likely not maintain good format documentation during any development cycle (or at least, not with our current amount of developer time available), only for stable releases. We also want the freedom to make breaking changes to formats during development cycles, so it seems like there is little point to writing good documentation because it will give people depending on the file formats for external tools a false sense of stability.

It makes sense that you don’t prioritize this. But, again, protos would give you this for free.

@craftyjon

Nobody has started on this and it’s an area where experimentation is lower-cost. I would suggest if you really want to push the idea of protos forward, this is where you start. We’d be happy to have your help with this if you’re up for it.

That’s good to know. I will look into this but probably after the plugin manager work I’m planning. And yes, I’m fully aware of the irony of myself being not core dev also having a bigger fish to fry