Lets not get hung up on 100,000 files per directory.... KiCAD currently doesn't have anywhere near that number of parts.
I agree. Lets not put arbitrary limits, anywhere.
But can we also agree on a duality that not only can a ".lib" file be a library, but also a directory (of parts or .lib files) be considered a library? In other words, intermixing .lib files and .kicad_mod part files into the same (bunch of) directories ( and getting rid of the ".pretty" directories).
Thinking Top-Down, Digikey says they have 6M parts, so a scheme should be able to handle at least that. Creating a main (ie the first major) library directory (say KiCAD-parts.hash), there could be a 1000-way hash (on the .lib or .part/.prt) filename ... with 1000 directories called lib.000, lib.001, .... lib.999 and Digikeys 6M parts would be distributed evenly (with a decent hash algorithm) with 6,000 parts in each directory (no problem for Linux/macOS/BSD). That is for storing all 6M parts, but realistically, KiCAD currently has (say) 1000 parts? For the current situation, with all 1000 parts hashed into 1000 directories, even Windoze should be able to handle it.
The issue is... how does a library or part (say abcd.lib or abcd.part) get into its correct hashed directory? I suggest putting it into the KiCAD.hash directory (along with the 1000 directories lib.000, lib.001, .... lib.999) and next time the part manager is started up, it discovers those loose ends, hashes their names, and places them into their proper hash directory. Hashing is easy and lightening-fast, but if that is a problem, I can provide such a hash-on-the-name algorithm.
How does a human find the file (for whatever purpose)? Just find the file using its name, and the appropriate OS command (ex: find -iname "abcd*" ).