README.md 2.51 KB
Newer Older
1
2
![ |small ](imgs/toolchain.png)

Jonathan Skone's avatar
Jonathan Skone committed
3
# Anaconda Python Environment Management Toolchain
4
5

This repository contains the tools used to manage the conda environments on RCC 
6
following the Python module restructure, which will take place September 3rd 2020
7
8
9
ahead of the next UofC quarter. The Conda environments are stored in YAML files 
that are managed by two Anaconda toolchain shell scripts `contractor.sh` and `architect.sh`

10
## The Anaconda Environment Toolchain scripts
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

### Architect -- [architect.sh](./architect.sh)
Script for keeping up to date list of RCC managed conda environments.
The architect has two purposes:

1. Inspecting the most recent Conda module's environments, and saving any recent changes by exporting the environment as a YAML file in $YAMLPATH/current

2. Backing up Conda environments from the old Anaconda3 modules, and saving them as YAML files to $YAMLPATH/archive. This does not need to be done more than once, provided no new environments are created using old Anaconda3 modules.

The YAMLPATH and other variables are configurable by editing the config.sh script. 
This script is run by a cron on the management node and will nightly update the list of conda environments in the system Anaconda python path.

### Contractor -- [contractor.sh](./contractor.sh)
Script for restoring Conda environments from the architect created YAML files.
The contractor has two basic uses:

1. Creating a particular environment from a YAML file in the repo. This is the default behavior of contractor.sh.
2. Migrating environments from one version of Anaconda to the next. This is done using the --restore flag.

Controlling which environments are restored is done by putting the YAML files to 
be restored under $YAMLPATH/current. To prevent an environment from being restored
in the next update, move its YAML file to $YAMLPATH/archive.
The architect.sh tool automatically creates YAML files in $YAMLPATH/current.

35
## Staff Instructions for Using Toolchain
36
37
38

In general the staff should not need to use the toolchain tools. The `architect.sh` 
script is run by a cron, but there may be the need for staff to use the `contractor.sh` 
Joseph Laurence Marques's avatar
Joseph Laurence Marques committed
39
40
script to restore conda environments (such as when a member of the staff installs a new 
version of Anaconda). Instructions for using this tool are in the 
41
42
43
44
45
[Staff Guide page](./docs/staffguide.md) of this repository.

## Conda Environment YAML files
The architect.sh script will nightly populate the `current` directory in this repo.
The `archive` folder contains all past conda environments that are no longer maintained.
46