Remove it whilst playing, it simply is not worth the risk. Sometimes even scrum caps come off during a game(happened a couple of times recently in 6 Nations games) and the potential for injury is greatly increased, not just to yourself but to other players too.
Take it off. Its the same like wearing a chain, just because you have a shirt on doesn't mean it can't be pulled off. Take it off.