TAU is a microtubule-associated protein that under pathological conditions such as Alzheimer's disease (AD) forms insoluble, filamentous aggregates. When 20 years after TAU's discovery the first TAU transgenic mouse models were established, one declared goal that was achieved was the modeling of authentic TAU aggregate formation in the form of neurofibrillary tangles. However, as we review here, it has become increasingly clear that TAU causes damage much before these filamentous aggregates develop. In fact, because TAU is a scaffolding protein, increased levels and an altered subcellular localization (due to an increased insolubility and impaired clearance) result in the interaction of TAU with cellular proteins with which it would otherwise either not interact or do so to a lesser degree, thereby impairing their physiological functions. We specifically discuss the non-axonal localization of TAU, the role phosphorylation has in TAU toxicity and how TAU impairs mitochondrial functions. A major emphasis is on what we have learned from the four available TAU knock-out models in mice, and the knock-out of the TAU/MAP2 homolog PTL-1 in worms. It has been proposed that in human pathological conditions such as AD, a rare toxic TAU species exists which needs to be specifically removed to abrogate TAU's toxicity and restore neuronal functions. However, what is toxic in one context may not be in another, and simply reducing, but not fully abolishing TAU levels may be sufficient to abrogate TAU toxicity.