You should be able to view your added 3D model; view -> 3D viewer
You can also directly add the 3D location and scaling to the mod file using a text editor. At the end of the file (without 3D defined) you can add the following (with your own file path and names)
(pad 3 thru_hole oval ........
(at (xyz 0 0 0))
(scale (xyz 0.3937 0.3937 0.3937))
(rotate (xyz 0 0 0))
Hopefully this should be useful to people wanting to add 3D models without mixing up the default librarys with custom parts.
UPDATE For those using nightly and upcoming version 5
Saving as a STEP format instead of VRML you need to make sure that the file suffix is *.STEP when exporting. As this ensures that the colour information is saved to file. You can then rename the file to *.stp which will allow you to import it into KiCad.
And then it imports without needing to apply scaling factors for mm
Using STEP you do not need to also have wrl format defined as this is generated by the KiCad internally. The advantage of using the STEP models is that you can then export the whole populated PCB in STEP format back out to use in your mechanical work.