I don’t know for PCB artwork, but for schematic, I usually manage the libraries in another repository, which is not part of the project. To open the project, you don’t even need the other libraries if you commit the -cache.lib library inside your project.
In fact, I would warn against commiting the libraries or even making them part of the project by a submodule, because that means that you are starting to change local copies of libraries and may end up having to merge multiple versions of a lib. In the case you appended new components to the libs in two different projects, you are sure to get a conflict, even for local mono-user projects…