This is the third year of the SARS-CoV-2 pandemic, and yet most children remain unvaccinated. COVID-19 in children manifests as mostly mild or asymptomatic, however high viral titers and strong cellular and humoral responses are observed upon acute infection. It is still unclear how long these responses persist, and if they can protect from re-infection and/or disease severity. Here, we analyzed immune memory responses in a cohort of children and adults with COVID-19. Important differences between children and adults are evident in kinetics and profile of memory responses. Children develop early N-specific cytotoxic T cell responses, that rapidly expand and dominate their immune memory to the virus. Children’s anti-N, but not anti-S, antibody titers increase over time. Neutralization titers correlate with N-specific antibodies and CD8+T cells. However, antibodies generated by infection do not efficiently cross-neutralize variants Gamma or Delta. Our results indicate that mechanisms that protect from disease severity are possibly different from those that protect from reinfection, bringing novel insights for pediatric vaccine design. They also underline the importance of vaccination in children, who remain at risk for COVID-19 despite having been previously infected.