You answered your own question quite well
All you need is patience and not brutal force, you will be able to tap the left over part of the woodruff key out. Once you have done that check the edges of the key seat. They can be sharp but should not look like the blade of a serrated blade of a bread knive. If the edge is looking bad you can go over it with a small diamond file but you should know what you are doing as you want the new key to survive an eternity
Replacing the seal seems to be a brilliant idea
