That is exactly why you should have one central place where you put your libraries. (can be a git repo connected via submodule as i suggested or a separate directory on your pc that itself can of course be a repo.)
The chache.lib file should be part of the original project (No one ever suggested anything different). So if you change a symbol or footprint in the library kicad does not quietly update the copies within your project.
You only need to merge if either multiple people work on the same libraries (should only be a problem for schematic libraries. Footprint libraries are one file per footprint.)
Or if you don’t commit (and push) your changes right after you made them. (You would also need to make changes to the library via a different project for that to be a problem. So you need to make two mistakes at the same time.)
I use exactly this setup for many projects, and there never was a problem. (I try to use atomic commits.)
Another way to get into trouble (only if multiple people work on the same library/project) is by forgetting to pull before you start to work.
By the way, merging kicad stuff is not as dangerous as it may seem.
Within the last week i needed to merge in at least 5 changes done by other people in our teams symbol libraries.
Merging a full project might be a bit more dangerous. So i would advice against that.
(If you commit and push often, the danger of that happening is quite small.)
tl;dr: If you use git as intended you should not get in trouble if you manage your libraries within a git repo. Not even if you share it as a submodule in multiple projects. To be on the safe site use atomic commits and push/pull often.