Your motor is never going to have nearly enough torque to tighten an ER nut.
Those motors are designed for High RPM, and Power = Torque * Angular velocity.
So if you have a 400W spindle that is designed for (a modest) 10000rpm then:
10000/602pi = 1047 [Rad/s]
400 / 1047 = 0.38 [Nm]
Maybe you can get twice that torque by pushing much more current through the motor (which is usually fine for a short time) but the core will then be saturated and it simply does not respond to stronger magnetic fields.
Recommended torque for an ER11 (upto 7mm shaft) is 24Nm.
Which means you’re more than an order of magnitude off and it’s never going to work.
That stepper motor thing you saw almost certainly also had a gearbox to get enough torque.
Once I did see a very clever home-built tool changer.
It had a mechanical blocking mechanism for the spindle and a wrench that could rotate around the place you normally hold it with your hands.
Then it used the (X, Y) motion to maneuver the ER collet into the wrench and then draw a circle around the hinge point.
Another possibility is to use shaft extensions. You can buy ER collet holders which also have an extra female thread on the other end. You can screw these on an ER spindle instead of the normal ER nut.
These tighten themselves on the ER thread, and exchanging them is quicker than doing it with the ER collets themselves. They are longer, so you loose a bit of stability, but you also gain work height