If you use CvPCB for footprint association, then the whole notion of a manufacturer or house part number in the symbol necessarily goes away. The manufacturer part number refers to a specific combination of symbol, footprint, tolerance and all sorts of other criteria. Change the footprint, you change the manufacturer part number, and that's why CvPCB is unusable.
I completely understand the effort to create a library of fully-qualified ("atomic") parts. Every employer I've worked for (been a few) has had a company-standard library. All parts have symbols, footprints, house part number, 3D model.
The first thing to understand is that you don't need a million parts in the library. You only need the parts you use. This means you can start small, adding only the parts you need for the first design. For the next design, if you need new parts, you create them, you add them to the library, done. And before you add them to the library, you vet the symbols, you make sure you can buy the parts, you put them through your documentation and release system. Thus, maintenance of the library isn't onerous.
Now, I am the first to admit that the hobbyist using Kicad for one design a year doesn't need the overhead of an ERP system. And that's why Kicad has the option to do CvPCB and why it doesn't have an embedded part number field in the symbols and all of that.