Yes, I agree with that.
To avoid confusion, it’s not really a “sheet symbol”, but just a sheet, (or sub sheet, hierarchical sheet), but it’s not a symbol. Symbols come from a library.
And selecting the sheet (or hovering over it) in the schematic editor and then press [del] really should be enough. (Apart from saving your schematic when you exit the schematic editor). Deleting the orphaned sheet from the project directory is optionally. As long as it is not mentioned in the project, KiCad just ignores it just like other files inside the project directory.
I recently (last week) had a rogue sheet in my project, and deleted it from the schematic without giving it much thought, and continued working. I assumed I created that sheet myself (it’s a fuzzy project, and I’ve got a fuzzy head).
But it is possible there are bugs around deleting sub sheets. Just now I created a 3rd level sub sheet in the second level of a hierarchy and deleted it again (Using KiCad V8.0.5), and it “just worked” for me. Apparently it did not work for you. To check whether this is a bug in KiCad, you can make a copy of your project, delete any “proprietary” stuff, zip it up and post it on the forum (you do need to gain some “trust” on this forum before you can upload here, see New Member Information )
Another option is to first backup, and then use a text editor to find references to the old / deleted sheet in your project files and delete them. But do note this is a last resort option. KiCad’s files are human readable, but they are not designed to be editable and KiCad is very fussy about syntax errors when opening it’s own files.