Python environments¶
Plenty of MagAO-X software is written in Python, so we need a consistent environment.
There are two config files in magao-x/config that control conda environments on AOC/RTC/ICC:
conda_env_base.yml
conda_env_pinned.yml
The first, conda_env_base.yml
, stores just the top-level dependencies (i.e. not everything they depend on) without versions. This makes it easier to track what we really depend on in case we need to help conda along with failing dependency resolution.
The second, conda_env_pinned.yml
, reflects the actual set of packages installed in the environment. This lets us recreate/update (nearly) identical environments on AOC/RTC/ICC.
Adding a package¶
If there’s a package you need for your work on the instrument machines, it’s a good idea to make it part of the environment so a rebuild is sure to install it.
Installation¶
Install as normal, preferring conda
packages if possible:
$ conda install foopkg
# *only* if it's not found:
$ pip install foopkg
Verify that it imports, works, etc.
Now when you do conda env export
, you should see an entry for your new package. Example:
$ conda env export
name: base
channels:
- conda-forge
- defaults
dependencies:
[...]
- foopkg=0.1
[...]
If it was installed via pip, the package will be in a sub-list under the pip:
heading.
Updating the environment template¶
To add it to the template, first pull any changes:
cd /opt/MagAOX/config
git pull
# resolve any conflicts
Then open /opt/MagAOX/config/conda_env_base.yml
. You’ll see something like conda env export
, but without the version numbers. Add your package name (but not version, unless you know what you’re doing) to the list, being careful to put it under the pip:
heading if that’s how it was installed.
Updating the pinned packages¶
Now, you need to update the pinned packages in /opt/MagAOX/config/conda_env_pinned.yml
.
$ conda env export > /opt/MagAOX/config/conda_env_pinned.yml
Storing in version control¶
$ cd /opt/MagAOX/config
$ git add conda_env_pinned.yml conda_env_base.yml
$ git commit -m "Added Python package foopkg to conda envs"
$ git push
Replicating across all the machines¶
If you just performed these steps on AOC, now you need to connect to ICC and RTC and run the following steps:
$ cd /opt/MagAOX/config
$ git pull
$ conda env update -f /opt/MagAOX/config/conda_env_pinned.yml
Updating conda
¶
If there are updates to conda
itself, it’ll probably tell you. You can run conda update -n base -c defaults conda
to update it, but be sure to follow the steps beginning at Updating the pinned packages to record the upgrade.
Performing a fresh conda install/upgrade¶
Generally only something to do if things are totally messed up, there’s a new version of the Python interpreter itself, or both.
Move
/opt/miniconda3
out of the way (i.e.mv /opt/miniconda3 /opt/miniconda3.bak
)Edit
/opt/MagAOX/source/MagAOX/setup/install_python.sh
and changeMINICONDA_VERSION="X-pyXX_X.Y.Z"
appropriately, and commit/push to version control.Run
bash /opt/MagAOX/source/MagAOX/setup/install_python.sh
to download and install the newconda
to/opt/miniconda3
with appropriate permissions
At this point you should log out and back in to reset any environment variables that were set by the old conda
.
If the Python version hasn’t increased¶
In most cases, the version of Python shipped with Miniconda hasn’t changed.
Run
bash /opt/MagAOX/source/MagAOX/setup/configure_python.sh
If the Python version has changed¶
The default configure_python.sh
would try to restore the pinned versions from conda_env_pinned.yml
, but would fail because of the Python version mismatch. Instead, you need to create the environment from conda_env_base.yml
and update conda_env_pinned.yml
yourself
conda env update -f /opt/MagAOX/config/conda_env_base.yml
conda env export > /opt/MagAOX/config/conda_env_pinned.yml
$ cd /opt/MagAOX/config $ git add conda_env_pinned.yml $ git commit -m "Updated pinned packages for conda upgrade" $ git push
You will also need to rerun some of the files in /opt/MagAOX/source/MagAOX/setup/steps
that install Python packages into the environment. A (possibly incomplete list):
cd /opt/MagAOX/source/MagAOX/setup/steps && \
bash install_purepyindi.sh && \
bash install_imagestreamio_python.sh && \
bash install_magpyx.sh && \
bash install_sup.sh
Replicate across all the machines¶
SSH to the other machines and:
Move
/opt/miniconda3
out of the way (i.e.sudo mv /opt/miniconda3 /opt/miniconda3.bak
)Update the environment files:
cd /opt/MagAOX/config && git pull
Update the MagAO-X source:
cd /opt/MagAOX/source/MagAOX && git pull
Install Python via miniconda:
bash /opt/MagAOX/source/MagAOX/setup/install_python.sh
Configure Python via conda environment files:
bash /opt/MagAOX/source/MagAOX/setup/configure_python.sh
Ensure all our custom packages get installed
cd /opt/MagAOX/source/MagAOX/setup/steps && \
bash install_purepyindi.sh && \
bash install_imagestreamio_python.sh && \
bash install_magpyx.sh && \
bash install_sup.sh