One of the most frequent symptoms of COVID-19 is the loss of smell and taste. Based on the lack of expression of the virus entry proteins in olfactory receptor neurons, it was originally assumed that the new coronavirus (severe acute respiratory syndrome coronavirus 2, SARS-CoV-2) does not infect olfactory neurons. Recent studies have reported otherwise, opening the possibility that the virus can directly infect the brain by traveling along the olfactory nerve. Multiple animal models have been employed to assess mechanisms and routes of brain infection of SARS-CoV-2, often with conflicting results. We here review the current evidence for an olfactory route to brain infection and conclude that the case for infection of olfactory neurons is weak, based on animal and human studies. Consistent brain infection after SARS-CoV-2 inoculation in mouse models is only seen when the virus entry proteins are expressed abnormally, and the timeline and progression of rare neuro-invasion in these and in other animal models points to alternative routes to the brain, other than along the olfactory projections. COVID-19 patients can be assured that loss of smell does not necessarily mean that the SARS-CoV-2 virus has gained access to and has infected their brains.