Testbed Operation

Welcome to the Comprehensive Adaptive Optics and Coronagraph Test Instrument (CACTI). Here we’ll show what you need to do for running CACTI remotely using the Virtual Machine, which operates very similarly to MagAO-X with some tweaks.

Diagram showing systems on the testbed

Diagram showing the relationships between systems on the XWCL testbed. Red indicates the AO control loop. Full-size

Sections TODO: Config, Python Scripts

General Safety

A few things to know for running CACTI safely:

  1. Do not power up the bench if there are thunderstorms. Ideally unplug the bench from the UPS in case of thunderstorms.

  2. Do not power up the deformable mirror if humidity is above 15%.

  3. Check the humidity reading about once every 30 minutes.

  4. If over the course of several days the humidity is slowly rising check the following things:

    • The airflow from the hose to the DM box

    • The dessicant attached to the air flow pipes.

  5. If restarting or shutting down with xctrl, first zero the DM and then release it.

Remote operation with a virtual machine (VM)

Create a virtual machine to operate CACTI remotely. The VM handles the control GUIs and the cameras.

Initial Setup

  1. You will need an SSH key in ECDSA or ED25519 format, and a user account in group magaox-dev on exao0 (a.k.a. TIC) with that key authorized

  2. Follow the directions in Remote operation up to Check connectivity to MagAO-X. (Since you will be connecting to the testbed rather than the instrument, we check that connection below.)

  3. Connect to the VM by running vagrant ssh in the MagAOX folder. Your shell prompt will change to:

    [vagrant@centos7] $
  4. Verify connectivity to exao0/TIC by connecting to it with ssh (entering yes if prompted to verify the host key):

    $ vagrant ssh
    Last login: Fri Jun 11 04:30:39 2021 from
    [vagrant@centos7 ~]$ ssh tic
    The authenticity of host 'exao0.as.arizona.edu (' can't be established.
    ECDSA key fingerprint is SHA256:7a6jFVxlsQG9X5ZVPX3IHTxmT2c+qKEcStlxbZp4I4g.
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added 'exao0.as.arizona.edu,' (ECDSA) to the list of known hosts.
    Last login: Wed Jun 23 12:27:16 2021 from example.com
    exao0:~ you$ logout
    Connection to exao0.as.arizona.edu closed.
    [vagrant@centos7 ~]$
  5. To get from the vagrant prompt back to your normal prompt, use exit or logout. Note that the virtual machine is still running in the background, and you can reconnect with vagrant ssh.

  6. Navigate to the vm/ folder created within your MagAOX/ folder. You need to replace the MagAO-X calib and config folders with CACTI’s calib and config.

    1. In terminal, go to the vm/ folder:

      $ cd vm/
    2. Delete the original MagAOX config and calib folders:

      $ rm -r calib config
    3. Clone the testbed_calib and testbed_config repositories from GitHub into calib/ and config/:

      $ git clone https://github.com/magao-x/testbed_calib calib
      $ git clone https://github.com/magao-x/testbed_config config

At this point, you should be ready to have regular virtual machine startup for CACTI.


These directions are not guaranteed to work unless your virtual machine setup is adjusted for CACTI.

  1. In terminal, go to the MagAOX directory:

    $ cd ~/your/path/to/MagAOX/
  2. Start up the virtual machine:

    $ vagrant up
    $ vagrant ssh
  3. One you are in the virtual machine, bring up xctrl:

    $ xctrl startup

    From here, this should start up the following tmux sessions:


Your VM is now connected to the testbed control computer and you can go about your work.

VM Commands

These commands allow you to turn on various GUIs through the VM.


Power Control GUI. Allows you to turn on and off the cameras, lasers, DM, etc.

To operate, use:

$ pwrGUI &

To power on a device, slide the bar from left to right. Simiarly, to power off a device, slide the bar from right to left.

Troubleshooting tips:

  1. Sometimes nothing shows up in pwrGUI. Exit the window and enter xctrl restart to reboot the tmux sessions. Running pwrGUI should be back to normal.


Real Time Image Viewer GUI. Allows you to view livestreams of the camera. A detailed explanation for rtimv can be found in the Cameras section.

There is a little bit of preparation work to do before running rtimv.

  1. Power on the cameras you want to use in pwrGUI.

  2. Initialize the milkzmqClient so rtimv can see them. You can do this with:

    $ milkzmqClient -p 9000 localhost <shmim-1> <shmim-2> ... &

where each <shmim> is a device (camera, DM channels). Load up all the cameras you want to use. For example,:

$ milkzmqClient -p 9000 localhost camlgsfp camzwfs camtip &

will initialize the cameras camlgsfp, camzwfs, and camtip. Note that dark frames are considered separate image streams on their own, so a complete set would include camlgsfp camlgsfp_dark camzwfs camzwfs_dark camtip camtip_dark.

The backgrounded job (because we used &) will keep outputting messages to the terminal, interfering with the appearance of the shell prompt, but you can hit Enter a few times to get the familiar [vagrant@centos7 ~]$ back.

Tip: If you forgot the & at the end of the command and the command line is hanging, you can press ctrl + z to go back to the command line and then enter bg to put milkzmqClient in the background.

Tip: You don’t have to have all the cameras loaded at once. You can run another instance of milkzmqClient for another camera without affecting a pre-existing instance.

Here is an example of milkzmqClient successfully loading for camlgsfp. The milkzmqClient: Connected to camlgsfp line indicates data should be flowing.

[vagrant@centos7 ~]$ milkzmqClient -p 9000 localhost camlgsfp &
[2] 6332
[vagrant@centos7 ~]$ N: 2
milkzmqClient: Beginning receive at tcp://localhost:9000 for camlgsfp
milkzmqClient: Connected to camlgsfp
 [ MILK_SHM_DIR ] '/milk/shm'
 [ MILK_SHM_DIR ] '/milk/shm'
 [ MILK_SHM_DIR ] '/milk/shm'
  1. Now you can run rtimv. There’s two ways you can do this.

    1. To see the camera GUI with the INDI connected display, use:

      $ rtimv -c rtimv_<camera-name>.conf &

      where <camera-name> is the name of the camera. For example if using camlgsfp,:

      $ rtimv -c rtimv_camlgsfp.conf &

      Note: A .conf file for this <camera-name> must exist for this to run. If it’s not present, contact Jared.

    2. If you are not interested in the INDI connected display, use:

      $ rtimv <camera-name> &

      and you should get the rtimv GUI with no notes on the sides.

Troubleshooting tips:

  1. Check that <camera-name> is powered on in pwrGUI.

  2. Check that INDI recognizes the camera. If the <camera-name>.fsm property in cursesINDI says NODEVICE, then it is not being detected. Try checking the USB connection.

  3. If all else fails, try resetting milkzmqClient:

    1. Kill the rtimv and milkzmqClient jobs. At the vm command line, enter jobs and you will see all the jobs running with a number associated with it.

      [vagrant@centos7 ~]$ jobs
      [1]   Running                 pwrGUI &
      [2]-  Running                 milkzmqClient -p 9000 localhost camlgsfp &
      [3]+  Running                 rtimv -c rtimv_camlgsfp.conf &

      To stop a job, enter kill %n where n is the number. In this example, you need to stop the milkzmqClient on 2 and the rtimv on 3.

      [vagrant@centos7 ~]$ kill %2
      [vagrant@centos7 ~]$ milkzmqClient: Disconnected from camlgsfp
      [2]-  Done                    milkzmqClient -p 9000 localhost camlgsfp
      [vagrant@centos7 ~]$ jobs
      [1]-  Running                 pwrGUI &
      [3]+  Running                 rtimv -c rtimv_camlgsfp.conf &
      [vagrant@centos7 ~]$ kill %3
      [vagrant@centos7 ~]$ jobs
      [1]-  Running                 pwrGUI &
      [3]+  Terminated              rtimv -c rtimv_camlgsfp.conf
      [vagrant@centos7 ~]$ jobs
      [1]+  Running                 pwrGUI &
    2. Reinitialize the milkzmqClient.

      [vagrant@centos7 ~]$ milkzmqClient -p 9000 localhost camlgsfp &
    3. Restart the vm_tic_milkzmq process in xctrl with xctrl restart vm_tic_milkzmq.

      [vagrant@centos7 ~]$ xctrl restart vm_tic_milkzmq
      Waiting for tmux session for vm_tic_milkzmq to exit...
      Waiting for tmux session for vm_tic_milkzmq to exit...
      Ended tmux session for vm_tic_milkzmq
      Session vm_tic_milkzmq does not exist
      Created tmux session for vm_tic_milkzmq
      Executed in vm_tic_milkzmq session: '/opt/MagAOX/bin/sshDigger -n vm_tic_milkzmq'
      [vagrant@centos7 ~]$ milkzmqClient: Connected to camlgsfp
       [ MILK_SHM_DIR ] '/milk/shm'
       [ MILK_SHM_DIR ] '/milk/shm'
       [ MILK_SHM_DIR ] '/milk/shm'

      Here we can see at the last 4 lines that camlgsfp is restarted in milkzmqClient.

    4. Start up rtimv like in the previous directions. The GUI should be outputting properly now.


Region of Interest GUI for rtimv. A detailed explanation for roiGUI functions can be found in the Cameras section.

To operate, use:

$ roiGUI <camera-name> &

where <camera-name> is the camera you want to edit the ROI for rtimv.

Basic operation for setting up the ROI box:

  1. In the rtimv window, hover a cursor where you want the center of the ROI box located.

  2. The bottom left corner of the rtimv window will be the X and Y pixel coordinates at the cursor.

  3. Note these values and input them into the X Center and Y Center targets in roiGUI.

  4. To set up the box size, you can use the cursor to go to the edge of your ROI in rtimv and do some quick math to determine how box the box size will be.

  5. Input these values in the Width and Height in roiGUI.

  6. At this point, a colored box will show up in rtimv. Play around with the settings to get the desired ROI.

  7. Once completed, click the set button and the rtimv window will change to the ROI.


DM Control GUI. Controls the 1K DM. Apply flats, clear channels, release DM.

IMPORTANT: Before powering the DM in pwrGUI and operating dmCtrlGUI, you must verify the 1K DM humidity is below 15%. See Humidity Sensing for instructions on checking the humidity.

To operate, use:

$ dmCtrlGUI dmkilo &

This will open a GUI window.

  1. Initialize the DM by clicking on the initialize at the top right. Sometimes, the GUI starts pre-initialized.

  2. To load a DM flat, choose which file you’d like from the top drop down menu.

  3. Click on set_flat to load the flat.

  4. When you are done using the 1K DM, please click on zero flat then release before powering it down in pwrGUI.

Commands run on exao0

To startup exao0, open a new terminal and ssh with your account into exao0. Always run it in xsup.

$ ssh exao0
$ su xsup

Startup and shutdown with xctrl

From here, you can start running the various processes with xctrl.

Humidity Sensing

The Arduino humidity sensor has been moved from corona to exao0. The humidity sensor is connected via USB to /dev/ttyACM0 which can be monitored with screen provided that you are in the dialout user group on exao0.

If you are not in the dialout group, get someone to do sudo gpasswd -a USERNAME dialout and log in again.

Open a separate terminal and log into exao0 with your account (not xsup).

If you are starting from a fresh (re)boot:

$ screen /dev/ttyACM0

If the session already exists (i.e. was disconnected without killing it):

$ screen -rd

The screen should now show a bunch of environmental monitoring information that looks like this:

Humidity: 10.70 %    Temperature: 22.80 *C 73.04 *F  Heat index: 21.41 *C 70.55 *F

Please actively check the humidity levels every 30 minutes or so.

Do not operate the 1K DM if the humidity is above 15%!!

If somoene else is viewing the humidity monitor, even if they are “detached” from screen, you won’t be able to open it until they have killed their screen session (after reattaching if needed).

To kill (exit) the humidity monitor: Ctrl + a, release, then “k”, then “y” to confirm.

  • This releases the device for other users.

To detatch: Ctrl + a, release, then “d”.

  • This makes it easy to reattach with screen -rd


Allows you to set exposure times, ROI, etc directly.

To start cursesINDI, enter it in the exao0 terminal when in xsup:

$ cursesINDI

For general use:

  1. Enter the name of device and it will search for it.

    • Tip: Sometimes there are multiple versions of the device. Add “.” at the end of your device name to minimize scrolling.

  2. Once at the list, curse over “target” in second to right hand column. Hit “e” for edit, enter a new number, and then “y” for yes.

  3. To exit, hit Ctrl + c.

Tip: You can also run cursesINDI in the virtual machine instead through xsup@exao0.

The Eye Doctor for CACTI

Consult The Eye Doctor for general information.

Run eye doctor in the exao0 terminal under xsup. The general form of the command is:

$ dm_eye_doctor <portINDI> <dmModes> <camera-name> <psf_core_radius_pixels> <modes_to_optimize> <amplitude_search_range> --skip 1

For example, if you want to run dm_eye_doctor for the 1K DM using the camlgsfp camera and correct the lower order modes, it would be:

$ dm_eye_doctor 7626 kiloModes camlgsfp 8 2...10 0.1 --skip 1

If you want to go on higher order modes, change the <modes_to_optimize> value:

$ dm_eye_doctor 7626 kiloModes camlgsfp 8 10...30 0.1 --skip 1

Once you have a DM flat that produces a nice PSF, you can save the flat with:

$ dm_eye_doctor_update_flat kilo

And it will save a new flat in the dmCtrlGUI list at the very bottom with the date stamped on it. To run the new flat, you need to update dmCtrlGUI to zero the flat, select the new flat, and then set it.

Running Python to control exao0

There are a myriad of commands you can use to do things with exao0, such as saving data and running control loops.

Running Jupyter Notebook (Python)

To access Jupyter Notebook from exao0, you need to ssh into exao0 with another terminal:

$ ssh -L 9990:localhost:9999 exao0

Note: if your computer has a different access code for getting into exao0, use that in place of the exao0 portion of the command above.

Once connected through ssh, you can navigate to localhost:9990 on your internet browser. This will open up the jupyter notebook directory page under xsup. If a password is required, ask someone who has access for it.

From here, you can create your own directories and jupyter notebooks to run your python code.

Saving Camera Images with magpyx

There’s two ways to save images from the cameras, either through cursesINDI or using the ImageStream function within the magpyx python code. This section will cover how to use magpyx. More information on magpyx can be found here.

In jupyter, you need to import ImageStream:

$ from magpyx.utils import ImageStream

To declare a camera, set the name of the camera in <camera-name>:

$ cam = ImageStream(<camera-name>)

From here, you can do multiple types of tasks with the camera. Commands for using ImageStream can be found in the source code. When you collect data for the camera, it will use the settings for that camera that have been declared in cursesINDI and roiGUI.

If you want to get the immediate frame, you can run:

$ image = cam.grab_latest()

If you want to get a cube of images for n number frames, you can do:

$ imagecube = cam.grab_many(n)

When you are done with the camera, please close it off:

$ cam.close()

Tips for running ImageStream:

  1. It’s generally better to leave the ImageStream on if you’re going to do multiple things instead of constantly opening and closing it.

  2. If you make a change on the ROI, you will need to close and re-open ImageStream for it to work. Otherwise, a segfault and no one likes that.

  3. If the camera isn’t actively collecting data, you can change the exposure time in cursesINDI and ImageStream will update to the new value.