Attempted manual back-annotation and resultant file hell

Layout
1

Hi.

If nothing else this might be a good cautionary tale to others, but maybe we can figure out some kind of workflow while we wait for the 6.0 release towards the end of the year.

I’m running KiCad 5.1.9 on Linux Mint 20, x86-64.

I have a hierarchical design (root sheet + two sheets inserted as modules). As you can imagine, the PCB layout, despite my best efforts, has the (usual) complete jumble of reference designators as compared to the placement coordinates on the board. In other packages, I would have done a back-annotation using the built in utilities, but given that in KiCad this feature is only in the nightly builds I decided to try and force the issue manually.

The process I chose was by dumping the component position file as a csv (File -> Fabrication Outputs -> Footprint Position File), bringing this into a spreadsheet, and sorting by refdes. By copying the refdes column, and adding two columns immediately to the right of that, “schnew” and “pcbnew”, I had a way to crudely track the refdes updates, and away I went updating the fields in each column as I updated the refdes in both the schematic and pcb, using the spreadsheet to ensure they matched.

After I was done, I tested the work by generating a netlist from the schematic and loading it into the board design. No errors, but there were some copper pours that were now flagged with ratsnest lines because the net assignments had changed. Re-assigning those fixed that, and the layout and schematic were now apparently in sync, with usable test point and component refdes numbering on the PCB, so I saved my files, closed KiCad and carried on with other things.

Then I opened KiCad and my pcb again a little while later, and all the refdes changes were gone. Interestingly, all the other changes, like refdes repositioning, character sizing, rotation, etc. were all still intact. Even more interestingly, the expected problem with copper pours did not reappear, even though, based on the net names that are shown, they should not be connecting.

Additionally, the schematic now fails to open with the following error message:

Error loading schematic files “[path-to-schematic]/schematic-filename.sch”.
expected single character token in input/source
“[path-to-schematic]/schematic-filename.sch”
line 216, offset 29

So I have three questions, to start:

  1. What happened to the refdes changes in PcbNew? There were no error messages, and it seemed happy enough to save and close the file with nothing untoward spewed to the kicad console.
  2. I was able to, once, get the schematic to open, by renaming the pcb so the new one had an extension of .kicad_pcb-wtf so I could rename .kicad_pcb-bak to .kicad_pcb. Again, no error messages, but now the schematic no longer opens, regardless of file renaming shenanigans. How is this possible?
  3. While I have backups, some of the other changes I’ve made to the files made them far more readable and usable, so I’m loathe to give up on the files. Are there any suggestions for recovery of at least of the schematic, even if the parts are no longer in sync? I’m ok with extreme measures such as blowing away all of the parts on the pcb, but leaving the copper on the board, and then importing the netlist and replacing the parts to suit the copper traces and pours…

There is no .cmp file in the directory, nor was one intentionally generated for this design.

There’s a whole bunch of other questions I have, but I think we should start there, before heading down the rabbit hole of how these files compare to database files, manipulating primary keys, etc.

Thanks in advance,
N

2

First of all, you could have done the renaming only in the schematic and then updated the PCB like with any other changes. Also there is no need for manual netlist export anymore, as the whole update procedure can be done via Update PCB from Schematic (which is located next to the Load netlist button in Pcbnew, also in the menu Tools both in Eeschema and Pcbnew and bound to the key F8 in Eeschema).

Now, to get on with your questions:

  1. We can’t know. Were you doing any strange things to the files? Editing them manually? Having multiple KiCAD instances open? Did your disk get corrupted? And I could go on and on, but the point is that this usually doesn’t (shouldn’t) happen.
  2. The PCB file doesn’t have anything to do with the schematic not opening. They are two totally independent files. Also your statement contradicts the previous and my final answer to the actual question part of this item would be: it isn’t.
  3. Read and try to understand the error message you get when opening the schematic. Open the .sch file with a text edior and look at what the error message points you to. (Hint: line 216, around character 29.)

I hope that my snark doesn’t get through too much, but your wall of text was a chore to get through (and part of me still wonders what masochistic tendecies do I have to have read it all …). The inconsistencies and long-windedness don’t help either. And given the lack of timely responses, I’d say others would agree.

3

I can comment on that: the OP wanted geographical annotation, i.e. ordering by the PCB layout component location. It can’t be done in the schematic.

Otherwise I agree. It’s difficult to see what actually is the problem. Maybe the schematic and the layout weren’t in sync to start with? Annotations and other changes should always be kept as “atomic” as reasonably possible and updated from schematic to layout immediately, otherwise there’s a risk of going out of sync and loosing some schematic/layout component connections.

Backannotation, namely the real one in 5.99, is a new feature and it’s possible there’s a bug. Therefore it would be important to find out the reason of the error if it’s caused by something done with KiCad. But it doesn’t help to point out an error in the file if it can’t be reproduced easily step by step.

There’s no need for an external netlist file, you should always use Tools->Update PCB from Schematic in KiCad internally.

You explained so many things which don’t belong to the normal workflow that it’s difficult to say in which phase the problem occurred.

Sometimes it helps to upload a zipped project here. Maybe someone takes time to look into it.

1 Like
4

Hi eelik

Yeah; there were definitely things I did outside of the normal workflow, to try and achieve my goals. Always a risk, but often a good learning opportunity. I admit that my post was indeed too long, so sorry to all of you that slogged through all that

I did a bunch more experimentation after posting that, using clones of that project directory. The interesting outcomes, comparing pre- and post-mangled files, and behaviours are as follows:

  1. the error message (on attempted reopening) points to a line that looks identical in both cases, so I guess just like a gcc error during make, it might indicate a line number but the problem could just as well like somewhere else
  2. if I clone the directory, nuke the board files, and just open the schematic (as if it was a design that had not been transitioned from schematic to layout), – there’s no problem, no error message; it just opens as one would expect
  3. if the board file was opened standalone, stripped of all components, and then saved, the schematic would again open without a problem

This seems to point to the sync issue between at least these two files. I guess kind of like having two tables in a database but having the primary and foreign keys, that would otherwise link them, out of whack. (oversimplified, but I think/hope the point carries)
–> So the big question is how are the two files linked, in KiCad, and in which file(s) are the (sorry for lack of a better term) keys stored, to accomplish this? I’m quite happy to read into the gory details if I know where to look in the documentation.

In case you were curious:
I was able to recover the important changes by stripping all the parts from the board shape and copper layers, and reloading the netlist. Because the latest schematic (now) had the geographically correct refdes numbering, it was simply a matter of plunking them onto the copper stubs that matched the footprints, so about an hour of work.

The biggest takeaways from my experimentation, and both of your replies, is that

  1. there is a method of linking the schematic and the PCB, that doesn’t use a netlist (I evidently missed that rather significant change in the move from 4.x to 5.x) and this is something I’ll look into
  2. don’t do that again :wink:

The design project is proprietary and can’t be uploaded, sorry.
-I’m currently thinking of some way that I could provide some useful data in case it might help with the 5.99 work, but I think I should know more about the structure of the various files, first. That might allow me to crank out a similar circuit and beat on it a bit to try and reproduce such errors, documenting the steps. Let me know if you have any thoughts on this.

Thanks,
N

5

Yes, it should do that. There’s no live connection between the schematic and the layout. You can open them independently and even when they are both open in the same project they are independent internally. All communication between them must be explicit. Therefore it doesn’t matter if you change one file, even make it garbage, until you try to sync them.

This is why I said that all changes should be small and simple before you update one file from the other. If you do changes (component changes) in both, save them and then try to update in one direction, you may get into problems. Or if you make several kinds of changes in the schematic – add new components, change annotations, replace symbols with new ones – before updating the layout you may be in trouble.

In more complicated cases it’s also important to understand how connections between schematic components and layout components are made. That’s what you noticed already. Read Update PCB from Schematic's match methods. (EDIT: here’s another explanation: https://gitlab.com/kicad/code/kicad/-/issues/4012#note_302985104 .)

1 Like
6

Hi eelik,

Thanks for those links – those give very succinct answers to some of my questions, one of them being the “path” stanza that I was seeing in the .sch file. I was puzzled by the “path” name, and your post in that first link explained it well.

As for the changes being small and simple – yeah, I would normally agree. However, this design needed to be documented for contract board stuffing and for support down the road, making that kind of geographic refdes numbering pretty much mandatory, given the parts count. Sadly, pretty much the opposite of a small and simple undertaking :wink:

Cheers,
N

7

I believe that this plugin works on 5.1.9 and might be worth a spin. As you know, geographical reannotation is included in the development build 5.99.

8

Hi John,

Yeah possibly would have, but IIRC it was only tested as a Windows build. Didn’t want to add another project to my queue, by building and testing on Linux, hence my push to do it another way.

Hopefully works for others :slight_smile:

-N

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.