Compute Environments

The Fornax Science Console offers several pre-installed environments. The specific environments that are available depend on the Base Environment (container image) you selected when starting your server. All container images use an x86_64 Ubuntu Linux system. Each comes pre-installed with JupyterLab extensions as well as one or more Python software environments. You can customize your experience by installing additional Python software in the pre-installed environments as well as creating new environments, installing additional JupyterLab extensions, and installing non-Python software. This page describes the details.

Working with Python Environments¶

python3 is the default Python environment. It has general astronomy and plotting software.

Each of the Fornax demo notebooks has its own environment with a name of the form py-{notebook-name} (e.g. py-light_curve_collector and py-multiband_photometry). Each environment has the packages required to run the notebook pre-installed (see View Pre-installed Software). When opening the notebook, the corresponding kernel should automatically start. You can also select it from the drop down kernel menu at the top-right of an open notebook.

To activate a specific pip-based environment (see Environment Types for details) from the terminal, run: source $ENV_DIR/{environment-name}/bin/activate. For example, to activate the py-light_curve_classifier environment, run:

source $ENV_DIR/py-light_curve_classifier/bin/activate

and the following to deactivate it:

deactivate

Environment Types¶

There are two types of Python environments, pip-based and conda-based.

pip-based

The pip-based environments use uv to manage the packages. These environments contain pip-installable packages and are used in most cases. The default environments are installed under $ENV_DIR. They are activated as indicated above with source $ENV_DIR/{env-name}/bin/activate.

conda-based

These use micromamba to manage the packages (similar to conda/mamba): The conda-based environments are used with packages that are not pip-installable. Examples of this include heasoft and ciao in the high-energy container image. These are activated with micromamba activate {env-name} and deactivated with micromamba deactivate. These are also installed under $ENV_DIR. You can use micromamba env list to list the environments.

Pre-installed Environments¶

The following environments are pre-installed:

python3

This is the default Python environment. It has general astronomy and plotting software.

py-{notebook-name}

Each of the Fornax demo notebooks has its own environment with a name of the form py-{notebook-name} (e.g. py-light_curve_collector and py-multiband_photometry). Each environment has the packages required to run the notebook pre-installed. When opening the notebook, the corresponding kernel should automatically start. You can also select it from the drop down kernel menu at the top-right of an open notebook.

heasoft, ciao, fermi, sas

Environments for high energy software that includes: heasoft, Chandra ciao, Fermi analysis software, and XMM-Newton SAS.

See View Pre-installed Software to learn about specific libraries each environment contains.

Select an Environment¶

Notebook: To activate a specific environment from a notebook, click on the name of the notebook’s current environment at the top right and then select your desired environment from the kernel drop down menu. If you open a Fornax demo notebook and get a popup window asking you to select a kernel, choose the kernel from the drop down menu with the same name as the notebook you are opening. If you open any other notebook and get a popup window asking you to select a kernel, python3 is usually the best choice.

Terminal: To activate a specific environment from the terminal, run: source $ENV_DIR/{environment-name}/bin/activate. For example, to activate the py-light_curve_classifier environment, run:

source $ENV_DIR/py-light_curve_classifier/bin/activate

and the following to deactivate it:

deactivate

Install Additional Software¶

To install additional Python software, you can either update an existing environment or create a new one.

Update an Existing Environment¶

To add packages to a currently installed environment, you install them with pip (or the faster uv pip) after activating the relevant environment.

• Inside a notebook running the relevant environment, run !uv pip install ..., passing the extra packaged needed.

• In the terminal, after activating the environment run: uv pip install ....

This should work for both pip and conda-based environments.

If you want to add a small number of packages to a built-in environment, however, you can follow these steps:

• From the terminal, activate the desired environment (see View Pre-installed Software).

• Add the packages with: uv pip install --target $USER_ENV_DIR/{env_name} package-1 package-2. Where {env_name} is the folder name of choice.

• Tell the environment about the new location:

  • In a terminal: export PYTHONPATH=$USER_ENV_DIR/{env_name}:$PYTHONPATH.

  • In a notebook, add the following at the top:

import sys, os
sys.path.insert(0, f"{os.environ['USER_ENV_DIR']}/{env_name}")

Create a New Environment¶

To create a new environment, we recommend using one of the provided scripts: setup-pip-env or setup-conda-env.

Run setup-pip-env -h or setup-conda-env -h from the terminal for detailed help. These scripts take either a requirements file (former) or a conda yaml file (latter), and create the environment, including the setup of the kernel so you can use the environment in a notebook.

• For pip-based environments (recommended):

  • Create a requirement file named: requirements-{env-name}.txt (e.g. requirements-myenv.txt bellow).

  • Call setup-pip-env from the same folder. By default, the environment is created in a global location ($ENV_DIR), that is reset at the start of every session. Use this for environments that are needed for a single session. If you want the environment to persist between sessions, use setup-pip-env --user. This will install the new environment under $USER_ENV_DIR (defaults to ~/user-envs).

numpy == 2.2.0
astropy

• For conda-based environment (If your packages are not pip-installable):

  • Create an environment file: conda-{env-name}.yml (e.g. conda-myenv.yml below).

  • Call setup-conda-env. Similar to the pip-case, the environment is created in a global default location ($ENV_DIR), that is reset at the start of every session. If you want the environment to persist between sessions, use setup-conda-env --user.

name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.11
  - numpy=2.2.0
  - pip
  - pip:
    - matplotlib

cd $USER_ENV_DIR
uv venv myenv --python=3.11
source myenv/bin/activate
# add numpy for example
uv pip install "numpy<2"

uv pip install ipykernel
python -m ipykernel install --name myenv --user

micromamba create -p ~/user-envs/my-conda-env python=3.12 pandas
micromamba activate -p ~/user-envs/my-conda-env

JupyterLab Extensions¶

Pre-installed extensions are described on the JupyterLab page.

Install a New Extension¶

Instructions on how to find and install extensions can be found at JupyterLab: Extensions. Extensions may include a front-end component, a server-side component, or both. You can install front-end extensions after JupyterLab starts, and they can show up if you refresh the page, as long they are installed in the environment running JupyterLab (/opt/jupyter/). Extensions that include a server-side component cannot be installed by individual users because they must be installed before JupyterLab starts. In that case, please open a helpdesk request on the Fornax Community Forum.

Compilers and General Software¶

As part of the system optimization and to allow for users to manage their own software, the list of packages installed in the system (using ubuntu apt) is kept to a minimum. Many of the useful packages (vim, htop, git, awscli, etc) are installed from conda-forge into the base conda environment under $ENV_DIR/base. You can add packages to this environment by doing:

micromamba install package_name

You can also include compilers. For example, to install C, C++ and Fortran compilers, you can do:

micromamba install c-compiler cxx-compiler fortran-compiler

For non-Python tools (e.g. htop, vim etc), they can be run directly from the terminal without a need for activating the base environment as they are included in the PATH by default.

Terminal Initialization scripts¶

The system uses bash as a default shell. The Jupyterlab terminal uses a non-login shell, which means ~/.bashrc is not called by default when a new terminal session starts. ~/.profile on the other hand is called. You can therefore use it for any bash initialization code. A new ~/.profile is created at login time if it does not exist, and it also calls ~/.bashrc, so you can add you customization (e.g. update PATH, setup rust or julia, etc) to either one.