Use one body per color and at the end use the boolean operators to combined the bodies. No need to work on a face by face basis. (I personally use the part workbench boolean here as the part design boolean did not work in the past. Not sure how well that one works now.)
A tip: do not use realistic colors in the design stage. Use bright easy do identify colors and then assign realistic colors when exporting using stepup.