Not sure about 2004, but on my 2014 i removed most of the side to side movement by pressing the left hand side bearings in a bit more (just a tad, maybe less than 1mm, if i press them more the PV get's stuck probably on the left exhaust valve).
p.s. Did you check the rocker arm on the right side or the bell crank hitting the case ? It's pretty common for them to bang on the cylinder or case, to check remove the right side PV cover, put the pipe back in, start the bike and rev it a tad, you will see the rocker arm moving and rattling, make sure it's not banging on the cylinder (if it's banging on the case, you should see marks on the case, you have to drain the coolant, the case oil can stay in just lean the bike on the left side when removing the case cover). Here is a video on youtube that helped me out a lot in removing most of my rattles at high rpms
https://www.youtube.com/watch?v=dI0fUUHaDjQ