The whole mess started with a command that should have been boring. On macOS, Miniconda's base environment was on conda 4.8.4, and conda update -n base conda --yes kept saying a newer version existed while refusing to install it:
==> WARNING: A newer version of conda exists. <==
current version: 4.8.4
latest version: 4.10.1
Then, at the end, the same command printed:
# All requested packages already installed.
That contradiction is where the afternoon went wrong.
I tried conda update python instead. The solver admitted the environment was inconsistent, then proposed a big jump: conda from 4.8.4 to 4.10.1, Python from 3.7.6 to 3.9.5, and a pile of libraries. It downloaded packages, executed the transaction, and then conda itself collapsed:
ModuleNotFoundError: No module named 'ruamel_yaml'
The next layer of the traceback was:
ImportError: No yaml library available.
To proceed, conda install ruamel_yaml
Conda's crash reporter then offered to send the report to the core maintainers. The upload did not complete. Of course it did not.
Running conda install ruamel_yaml produced the same YAML error, because conda needed the YAML library to start and could not install the YAML library without starting. I checked what was actually on PATH:
$ which pip
/opt/miniconda3/bin/pip
$ which python
/opt/miniconda3/bin/python
$ python --version
Python 3.9.5
$ conda --version
conda 4.10.1
So I went around conda with pip:
$ pip install ruamel.yaml
That worked:
Successfully installed ruamel.yaml-0.17.9 ruamel.yaml.clib-0.2.2
Retrying conda install ruamel_yaml got past YAML and hit the next missing piece:
ModuleNotFoundError: No module named 'conda_package_handling'
Both conda install conda_package_handling and conda update -n base failed on the same import. I found a report from someone who had recovered by copying an older conda_package_handling from another machine and then running conda install conda-package-handling. Luckily, /opt/miniconda3/lib/python3.7/site-packages/conda/ still had conda_package_handling and conda_package_handling-1.6.0.dist-info. I copied both into /opt/miniconda3/lib/python3.9/site-packages/conda/.
That got conda further, but not all the way:
CondaDependencyError: Cannot run solver. No functioning SAT implementations available.
The SAT solver was pycosat, so I tried pip install pycosat --force-reinstall. The first attempt died on:
ModuleNotFoundError: No module named 'setuptools'
I copied files around and tried again. The second attempt died on:
ModuleNotFoundError: No module named '_distutils_hack'
The third attempt died on:
ModuleNotFoundError: No module named 'pkg_resources'
After more find-copy-paste, the fourth attempt finally built and installed:
Successfully built pycosat
Successfully installed pycosat-0.6.3
Pip also warned:
conda 4.10.1 requires ruamel_yaml_conda>=0.11.14, which is not installed.
But by then conda could run again. conda update conda completed, though it downgraded libblas, libcblas, libgfortran, liblapack, and libopenblas, and updated sqlite. A later conda update --all did little more than downgrade five packages. After that afternoon, I was past being surprised by Python.
There is one useful thing I learned: conda keeps revisions. conda list --revision showed a working state before the upgrade:
2021-06-10 19:28:33 (rev 37)
and the broken upgrade:
2021-06-10 19:35:15 (rev 38)
conda {4.8.4 (conda-forge/osx-64) -> 4.10.1 (conda-forge/osx-64)}
python {3.7.6 (conda-forge/osx-64) -> 3.9.5 (defaults/osx-64)}
In theory, conda install --revision=37 would have rolled the environment back to the last working version. That is good to know for next time, if there is a next time.
What made the whole thing worse was how familiar it felt. The issue threads around this kind of failure are full of users helping each other through missing modules and broken transactions, often with little more than a workaround. The lesson looks less like 'read the docs' and more like 'do not use conda.' Or maybe do not use Python. Or, if you are not ready to spend two and a half hours debugging one environment line by line, do not get into programming at all.