This is really not new, but it does not seem to be widely known, so here for future reference maybe it helps someone. Turns out it is quite easy to free up space with anacondas own tools:
conda clean --all --dry-run
will show you all that could be removed and how much space will be reclaimed. Removing the "--dry-run" from the command will actually run it and delete the cached files.
Background:
I was wondering why my disk was filling up and why anaconda used so much diskspace (found out using QDirStat, same tool exists for Windows (WinDirStat) and Mac).
Of course anaconda “copies” and keeps different versions in the different environments you create - that is on purpose and by design and has to be accepted. But it turns out anaconda “caches” all packages it ever downloads. That means I had 10 different old versions of pytorch, tensorflow, mkl etc. tarballs, each using several hundred MBs.
Up until now I always “reclaimed” that space by manually deleting stuff, but that sometimes caused errors later on when trying to install new envs, as something anaconda thought was cached had been removed manually. Who knew it was so easy to make conda handle that for us 
!