Write it down on your notepad. Referees should keep a notepad in their pocket to keep score and other notes such as bookings and what not. Hope this helps
You can have a small notebook with a pencil to keep track of the score. You can also use the notebook to write player's names down who received a yellow card or was sent off etc
Team managers keep the scores as well as the scorers.