I used one of the Dremel bits shaped like this photo to plunge cut out the remaining original plug. Be careful not to go to deep ;-) This is not easy because the bit wants to grab the sides and eat the hole to a larger diameter. It almost makes sense to grid off the side cutting flutes so you only have the plunge / face cutting teeth.
https://www.homedepot.com/p/Dremel-5...-115/203672169