I wouldn't lose any sleep over it either
It's not like it will be cause for the new bearing not to seat properly. I figure if you pressed out and in enough times you could run into issues but probably still not likely.
When I did mine, even with heat I couldn't get the needle rollers for the transmission shafts to come out, and no chance to use a punch from the far side on one of them. Had to get to it with a blind bearing puller.
Even without dry ice, new bearings will easily drop into a hot set of cases as long as you can get the bearing square before letting go of it.