This is because that is a system path. You need root access for it. (In other words: Do not touch this if you are not 100% certain you know what you are doing.)
Your personal libs should not be in this path!! (These libs will be managed by the kicad package.)
Running kicad without libs is done by removing footprint libs from the fp lib table (library manager in pcb_new). For symbols such an option does not exist in version 4. Symbol libs are assigned to the project. So you can only influence what libs are assigned to new projects (My suggestion: Do not start new projects with version 4. Switch to version 5 for new stuff.)
But if you insist on creating new projects with version 4 then you might need to edit the system template project to remove symbol libs for new projects.
Or you simply take care of library preiorities and make sure your personal symbol libs have higher priority than the system ones: Any conflict with names of modules and libraries? - #2 by Rene_Poschl This should remove any conflict between your personal libs and the system libs as your lib takes precedence in the case of a conflict.