The correct way is to use oval holes where you need an oval cutout. More complex coutouts will indeed require the use of an additional layer or a way to create complex hole shapes in some other way. You might want to create a bug report for the later. (Kicad will support the edge cuts layer in the future but as i said this workaround is no longer possible.)
Assuming the footprint have Cut layer, What if I place a filled rectangular in copper layer overlay it. Will the clearance rule will remove it away from the line on Cut layer, or will it only complain in DRC check? What is your guess?