Battery holder in which coin cell can be inserted vertically