Courtyard is the area around a physical component required for assembly and other functional purposes. No other component should be placed inside that area. If components are too close to each other, assembly may be difficult or impossible or the component may not behave properly.
When discussing assembly, this really depends on the assembly style (automatic or manual) together with other possible details. There is no one correct area, shape or dimensions of the courtyard which would guarantee problem free assembly, while still being optimally small, to allow dense placement when needed. It’s always a best guess compromise.
As a crude guide to check that components aren’t too close together, KiCad can check courtyard collisions. This checks overlapping courtyard areas. For this check, the area must be a closed shape. The shape is drawn graphically on the F.Courtyard or B.Courtyard layer. The requirements are exactly the same as with Edge.Cuts board edge: each line must end where the next one starts, they must not cross and the last one must end where the first one starts. There must nothing else on that layer in the footprint other than this closed shape.
A rectangle is often used as a courtyard, and, in most cases, one rectangle shape should be sufficient.
If a courtyard drawing doesn’t obey these rules or there exists other problems with a courtyard, KiCad can give an error:
Error: Footprint has malformed courtyard (not a closed shape)
Error: Footprint has malformed courtyard (self-intersecting)
Both above examples of errors show a courtyard not following these rules.
Violation severities can be altered in the Board Setup.
How important are courtyards and their severities?
Personal decisions need to be made. A footprint from a 3rd party may have anything in the courtyard. Some official KiCad libraries, as this is being written, have malformed or guessed courtyards even when they appear to be correct.
If you design or modify your own footprints you can draw a courtyard which is good enough for your purposes, even though there just aren’t any hard rules to follow. The best advice is probably found in the KiCad Library Conventions (used for the official libraries) in F5.3 Courtyard layer requirements - Library Conventions | KiCad EDA. If you want to optimize, you would need to know the limits of the assembly process and the interplay between that component and surrounding components. For example: imagine a very tall and a very short component standing side by side: the tall may be placed easily, but assembling the short may be problematic as a result of the tall placement, even when the courtyards are estimated to be enough in their own right.
Enough room for manufacture needs to be left around components. If a tighter design is required, decisions, on a case by case basis, need to be made as to whether some courtyards can overlap other courtyards. Violations between two components in the DRC can then be either excluded or ignored.
Notice that although logically, in an ideal case with ideal courtyards, each component only requires the space around it formed by its own courtyard, overlapping courtyards are allowed. The DRC checks this overlap. However, although the courtyards of two components can be allowed to overlap, this overlap should exclude conditions where one component’s pads, or possibly outline (ideally drawn on the Fab layer), overlap another component’s courtyard.
In the real world, the modern automatic assembly processes can do quite dense placements. In my personal experience I have seen that, for example, with 0402 components, the copper clearances limit placement more than the suggested courtyards by, say, the official KiCad footprints.