If you watch professional or international matches, you'll see plenty of front and second rowers who do tape their ears. You'll also see others who wear a combination of bandages, medical and electrical on the ears, heads, thighs ... whatever. And if you want to see something truly remarkable, check out the photos of the contraption Imanol Harinordoquy wore during an Heineken Cup match a few years ago to protect a broken nose. Do whatever you think is best -- it's nobody else's concern.
You'll look a lot less stupid wearing tape than you would with cauliflower ears.
Yeah go for it, you might get the odd question of some of the lads in your team, but thats it. At the end of the day, you do not want cauliflower ears!