Welcome to the Comprehensive Adaptive Optics and Coronagraph Test Instrument (CACTI). Here we’ll show what you need to do for running CACTI remotely using the Virtual Machine, which operates very similarly to MagAO-X with some tweaks.
Sections TODO: Config, Python Scripts
A few things to know for running CACTI safely:
Do not power up the bench if there are thunderstorms. Ideally unplug the bench from the UPS in case of thunderstorms.
Do not power up the deformable mirror if humidity is above 15%.
On humid (monsoon) days, it’s recommended to check the humidity reading about once every 30 minutes.
If over the course of several days the humidity is slowly rising check the following things:
The airflow from the hose to the DM box
The dessicant attached to the air flow pipes.
If restarting or shutting down with
xctrl, first zero the DM and then release it.
Here are some resources for troubleshooting things that aren’t working:
For the most part, troubleshooting tips are documented with the command. It’s not an exhaustive list, though.
General and sepcific troubleshooting tips can be found in the Troubleshooting page.
If you’re not sure how to proceed, please bring up the issue in the
Remote operation with a virtual machine (VM)¶
Create a virtual machine to operate CACTI remotely. The VM handles the control GUIs and the cameras.
You will need an SSH key in ECDSA or ED25519 format, and a user account in group
exao0(a.k.a. TIC) with that key authorized
Connect to the VM by running
vagrant sshin the MagAOX folder. Your shell prompt will change to:
Verify connectivity to
exao0/TIC by connecting to it with ssh (entering
yesif prompted to verify the host key):
$ vagrant ssh Last login: Fri Jun 11 04:30:39 2021 from 10.0.2.2 [vagrant@centos7 ~]$ ssh tic The authenticity of host 'exao0.as.arizona.edu (126.96.36.199)' can't be established. ECDSA key fingerprint is SHA256:7a6jFVxlsQG9X5ZVPX3IHTxmT2c+qKEcStlxbZp4I4g. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'exao0.as.arizona.edu,188.8.131.52' (ECDSA) to the list of known hosts. Last login: Wed Jun 23 12:27:16 2021 from example.com exao0:~ you$ logout Connection to exao0.as.arizona.edu closed. [vagrant@centos7 ~]$
To get from the vagrant prompt back to your normal prompt, use
logout. Note that the virtual machine is still running in the background, and you can reconnect with
Navigate to the
vm/folder created within your
MagAOX/folder. You need to replace the MagAO-X
configfolders with CACTI’s
In terminal, go to the
$ cd vm/
Delete the original MagAOX config and calib folders:
$ rm -r calib config
Clone the testbed_calib and testbed_config repositories from GitHub into
$ git clone https://github.com/magao-x/testbed_calib calib $ git clone https://github.com/magao-x/testbed_config config
At this point, you should be ready to have regular virtual machine startup for CACTI.
Updating the VM¶
When testbed code is updated and pushed onto Github, your computer cannot register them until you update
your virtual machine. Ideally, you should update both
In terminal, go to the MagAO-X directory:
$ cd ~/your/path/to/MagAOX/
Go to the
$ cd vm/
Go to the
config/folder and go
$ cd config/ $ git pull
From here, you will see some text about files being downloaded and updated. If there is nothing new, then you will get an
Already up to date.response.
You can update
calib/by moving up to
vm/and following step 3:
$ cd .. $ cd calib/ $ git pull
After this, your virtual machine will be up to date with the new CACTI code.
These directions are not guaranteed to work unless your virtual machine setup is adjusted for CACTI.
In terminal, go to the MagAOX directory:
$ cd ~/your/path/to/MagAOX/
Start up the virtual machine:
$ vagrant up $ vagrant ssh
One you are in the virtual machine, bring up xctrl:
$ xctrl startup
From here, this should start up the following tmux sessions:
vm_tic_indi toc_tic_milkzmq toc_tic_jupyterlab
Your VM is now connected to the testbed control computer and you can go about your work.
These commands allow you to turn on various GUIs through the VM.
Power Control GUI. Allows you to turn on and off the cameras, lasers, DM, etc.
To operate, use:
$ pwrGUI &
To power on a device, slide the bar from left to right. Simiarly, to power off a device, slide the bar from right to left.
Sometimes nothing shows up in
pwrGUI. Exit the window and enter
xctrl restartto reboot the tmux sessions. Running
pwrGUIshould be back to normal.
Sometimes only one row shows up (either
usb). Here, we have to go on
TICto investigate. This will be a very simplified explanation from xctrl section, which contains more details on other troubleshooting instances.
TICfrom within Vagrant:
$ ssh tic
xctrl statusto investigate the various processes. Green means the process is working. Red means the process has some problems with it. Other modes are explained in xctrl:
$ xctrl status
For example, say the
usbdu0process is highlighted in red. You can restart it with
xctrl startup usbdu0:
$ xctrl startup usbdu0
Check that it got fixed with the
usbdu0process highlighted in green with
$ xctrl status
You should be good to go. Return to Vagrant by typing
If you still have connection problems with Vagrant, exit out of Vagrant and type
vagrant reloadin the terminal. This will restart your entire virtual machine. This works best when you’ve left your computer on for many days without restarting.
Real Time Image Viewer GUI. Allows you to view livestreams of the camera. A detailed
rtimv can be found in the Cameras
There is a little bit of preparation work to do before running
Power on the cameras you want to use in
rtimvcan see them. You can do this with:
$ milkzmqClient -p 9000 localhost <shmim-1> <shmim-2> ... &
<shmim>is a device (camera, DM channels). Load up all the cameras you want to use. For example,:$ milkzmqClient -p 9000 localhost camlgsfp camzwfs camtip &
will initialize the cameras
camtip. Note that dark frames are considered separate image streams on their own, so a complete set would include
camlgsfp camlgsfp_dark camzwfs camzwfs_dark camtip camtip_dark.
The backgrounded job (because we used
&) will keep outputting messages to the terminal, interfering with the appearance of the shell prompt, but you can hit
Entera few times to get the familiar
Tip: If you forgot the
&at the end of the command and the command line is hanging, you can press
ctrl + zto go back to the command line and then enter
milkzmqClientin the background.
Tip: You don’t have to have all the cameras loaded at once. You can run another instance of
milkzmqClientfor another camera without affecting a pre-existing instance.
Here is an example of
milkzmqClientsuccessfully loading for
milkzmqClient: Connected to camlgsfpline indicates data should be flowing.[vagrant@centos7 ~]$ milkzmqClient -p 9000 localhost camlgsfp &  6332 [vagrant@centos7 ~]$ N: 2 camlgsfp milkzmqClient: Beginning receive at tcp://localhost:9000 for camlgsfp milkzmqClient: Connected to camlgsfp [ MILK_SHM_DIR ] '/milk/shm' [ MILK_SHM_DIR ] '/milk/shm' [ MILK_SHM_DIR ] '/milk/shm'
Now you can run
rtimv. There’s two ways you can do this.
To see the camera GUI with the INDI connected display, use:
$ rtimv -c rtimv_<camera-name>.conf &
<camera-name>is the name of the camera. For example if using
$ rtimv -c rtimv_camlgsfp.conf &
.conffile for this
<camera-name>must exist for this to run. If it’s not present, contact Jared.
If you are not interested in the INDI connected display, use:
$ rtimv <camera-name> &
and you should get the
rtimvGUI with no notes on the sides.
<camera-name>is powered on in
Check that INDI recognizes the camera. If the
NODEVICE, then it is not being detected. Try checking the USB connection.
If all else fails, try resetting
milkzmqClientjobs. At the vm command line, enter
jobsand you will see all the jobs running with a number associated with it.
[vagrant@centos7 ~]$ jobs  Running pwrGUI & - Running milkzmqClient -p 9000 localhost camlgsfp & + Running rtimv -c rtimv_camlgsfp.conf &
To stop a job, enter
nis the number. In this example, you need to stop the
milkzmqClienton 2 and the
[vagrant@centos7 ~]$ kill %2 [vagrant@centos7 ~]$ milkzmqClient: Disconnected from camlgsfp - Done milkzmqClient -p 9000 localhost camlgsfp [vagrant@centos7 ~]$ jobs - Running pwrGUI & + Running rtimv -c rtimv_camlgsfp.conf & [vagrant@centos7 ~]$ kill %3 [vagrant@centos7 ~]$ jobs - Running pwrGUI & + Terminated rtimv -c rtimv_camlgsfp.conf [vagrant@centos7 ~]$ jobs + Running pwrGUI &
[vagrant@centos7 ~]$ milkzmqClient -p 9000 localhost camlgsfp &
xctrl restart vm_tic_milkzmq.
[vagrant@centos7 ~]$ xctrl restart vm_tic_milkzmq Waiting for tmux session for vm_tic_milkzmq to exit... Waiting for tmux session for vm_tic_milkzmq to exit... Ended tmux session for vm_tic_milkzmq Session vm_tic_milkzmq does not exist Created tmux session for vm_tic_milkzmq Executed in vm_tic_milkzmq session: '/opt/MagAOX/bin/sshDigger -n vm_tic_milkzmq' [vagrant@centos7 ~]$ milkzmqClient: Connected to camlgsfp [ MILK_SHM_DIR ] '/milk/shm' [ MILK_SHM_DIR ] '/milk/shm' [ MILK_SHM_DIR ] '/milk/shm'
Here we can see at the last 4 lines that
camlgsfpis restarted in
rtimvlike in the previous directions. The GUI should be outputting properly now.
Region of Interest GUI for
rtimv. A detailed explanation for
roiGUI functions can be found
in the Cameras section.
To operate, use:
$ roiGUI <camera-name> &
<camera-name> is the camera you want to edit the ROI for
Basic operation for setting up the ROI box:
rtimvwindow, hover a cursor where you want the center of the ROI box located.
The bottom left corner of the
rtimvwindow will be the X and Y pixel coordinates at the cursor.
Note these values and input them into the
Y Centertargets in
To set up the box size, you can use the cursor to go to the edge of your ROI in
rtimvand do some quick math to determine how box the box size will be.
Input these values in the
At this point, a colored box will show up in
rtimv. Play around with the settings to get the desired ROI.
Once completed, click the
setbutton and the
rtimvwindow will change to the ROI.
DM Control GUI. Controls the 1K DM. Apply flats, clear channels, release DM.
IMPORTANT: Before powering the DM in
pwrGUI and operating
dmCtrlGUI, you must verify the
1K DM humidity is below 15%. See Humidity Sensing for instructions on checking the humidity.
To operate, use:
$ dmCtrlGUI dmkilo &
This will open a GUI window.
Initialize the DM by clicking on the
initializeat the top right. Sometimes, the GUI starts pre-initialized.
To load a DM flat, choose which file you’d like from the top drop down menu.
set_flatto load the flat.
When you are done using the 1K DM, please click on
releasebefore powering it down in
Sometimes the GUI claims the DM is off, despite it being powered on in
pwrGUI. Here’s some steps to take to investigate:
Verify that the power is working. Go on
pdu0.to the search portion. Scroll down until you find
dmkiloin the second column. Check that the
Onin the rightmost column.
Restart the driver process. Log into
xctrl status. Look for the
dmkilosession on the list. If it is red, then the process is not running. If it’s green, then the process is running. Either way, reset the driver with:
$ xctrl restart dmkilo
From here, this will reconnect the device and
dmCtrlGUIshould be active again.
DM Modes GUI. You can apply up to +/-1 wavelength of low order Zernike modes. Useful for manual dialing.
To operate, use:
$ dmModeGUI kiloModes &
This will open a GUI window with a list of low order Zernikes and a slider bar.
You can manually enter a number in the box at the right side, then hit enter to apply it.
You can move the slider bar to apply aberration.
You may see some values already in place. They are retained in the virtual machine.
Commands run on
To startup exao0, open a new terminal and ssh with your account into exao0. Always run it in xsup.
$ ssh exao0 $ su xsup
Startup and shutdown with
From here, you can start running the various processes with xctrl.
The Arduino humidity sensor has been moved from
exao0. The humidity
sensor is connected via USB to
/dev/ttyACM0 which can be monitored with
provided that you are in the
dialout user group on
If you are not in the
dialout group, get someone to do
sudo gpasswd -a USERNAME dialout
and log in again.
Open a separate terminal and log into
exao0 with your account (not
If you are starting from a fresh (re)boot:
$ screen /dev/ttyACM0
If the session already exists (i.e. was disconnected without killing it):
$ screen -rd
The screen should now show a bunch of environmental monitoring information that looks like this:
Humidity: 10.70 % Temperature: 22.80 *C 73.04 *F Heat index: 21.41 *C 70.55 *F
Please actively check the humidity levels every 30 minutes or so.
Do not operate the 1K DM if the humidity is above 15%!!
If somoene else is viewing the humidity monitor, even if they are “detached” from
you won’t be able to open it until they have killed their screen session (after reattaching if needed).
To kill (exit) the humidity monitor:
Ctrl + a, release, then “k”, then “y” to confirm.
This releases the device for other users.
Ctrl + a, release, then “d”.
This makes it easy to reattach with
Allows you to directly set exposure times, investigate the status of various components, etc.
To start cursesINDI, enter it in the
exao0 terminal when in
For general use:
Enter the name of device and it will search for it.
Tip: Sometimes there are multiple prefix versions of the device (such as camera darks). Add “.” at the end of your device name to minimize scrolling.
Once at the list, curse over “target” in second to right hand column. Hit “e” for edit, enter a new number, and then “y” for yes.
To exit, hit
Ctrl + c.
Tip: You can also run
cursesINDI in the virtual machine instead through
The Eye Doctor for CACTI¶
Consult The Eye Doctor for general information.
Before running eye doctor, make sure that the PSF on the camera is not oversaturated or else the solution
will not turn out well. Go on
cursesINDI to adjust the camera exposure time or add an ND filter to
Run eye doctor in the
exao0 terminal under
xsup. The general form of the command is:
$ dm_eye_doctor <portINDI> <dmModes> <camera-name> <psf_core_radius_pixels> <modes_to_optimize> <amplitude_search_range> --skip 1
For example, if you want to run
dm_eye_doctor for the 1K DM using the
camlgsfp camera and
correct the lower order modes, it would be:
$ dm_eye_doctor 7626 kiloModes camlgsfp 8 2...10 0.1 --skip 1
If you want to go on higher order modes, change the
$ dm_eye_doctor 7626 kiloModes camlgsfp 8 10...30 0.1 --skip 1
Once you have a DM flat that produces a nice PSF, you can save the flat with:
$ dm_eye_doctor_update_flat kilo
And it will save a new flat in the
dmCtrlGUI list at the very bottom with the date stamped on it.
To run the new flat, you need to update
dmCtrlGUI to zero the flat, select the new flat, and then
Note: Occasionally, Eye Doctor doesn’t give you the best solved flat. This may be remedied by going
dmModeGUI to manually dial out the lower order modes.
Running Python to control
There are a myriad of commands you can use to do things with
exao0, such as saving data and
running control loops.
Running Jupyter Notebook (Python)¶
To access Jupyter Notebook from
exao0, you need to ssh into
exao0 with another terminal:
$ ssh -L 9990:localhost:9999 exao0
Note: if your computer has a different access code for getting into
exao0, use that in
place of the
exao0 portion of the command above.
Once connected through ssh, you can navigate to
localhost:9990 on your internet browser. This
will open up the jupyter notebook directory page under
xsup. If a password is required, ask
someone who has access for it.
From here, you can create your own directories and jupyter notebooks to run your python code.
Saving Camera Images with
There’s two ways to save images from the cameras, either through
cursesINDI or using the
ImageStream function within the
magpyx python code. This section will cover how to use
magpyx. More information on
magpyx can be found here.
In jupyter, you need to import
$ from magpyx.utils import ImageStream
To declare a camera, set the name of the camera in
$ cam = ImageStream(<camera-name>)
From here, you can do multiple types of tasks with the camera. Commands for using
can be found in the source code.
When you collect data for the camera, it will use the settings for that camera that have been declared
If you want to get the immediate frame, you can run:
$ image = cam.grab_latest()
If you want to get a cube of images for
n number frames, you can do:
$ imagecube = cam.grab_many(n)
When you are done with the camera, please close it off:
Tips for running
If you want to save an image with
cam.grab_latest()after a change occured, a short delay before grabbing the frame is required. Here is some example code for setting a delay:
$ import time $ # do a command here $ time.sleep(0.5) # in seconds; can be shorter pending on camera exposure time $ image = cam.grab_latest()
It’s generally better to leave the
ImageStreamon if you’re going to do multiple things instead of constantly opening and closing it.
If you make a change on the ROI, you will need to close and re-open
ImageStreamfor it to work. Otherwise, a segfault and no one likes that.
If the camera isn’t actively collecting data, you can change the exposure time in
ImageStreamwill update to the new value.
Applying DM commands with
If you would like to apply DM commands through python, you can do so using
as saving a camera image with some simple changes. The main difference is that you will need to initialize
ImageStream for each DM channel you want to access. What shows up on the DM is the summed command
across each of the channels.
In jupyter, you need to import
$ from magpyx.utils import ImageStream
To declare the DM channel, set the name of the DM and channel in
$ dm = ImageStream(<dm-channel-name>)
<dm-channel-name> follows a specific format of
XX is the DM number
YY is the channel number.
On CACTI, there are two DM options that are available to use.
The BMC 1K is
00, making it
If the IrisAO segmented DM is in use, it would be
Despite different DMs, they each have 12 channels available for use. However, note that some channels are used for specific activities in CACAO. Here are some channels generally used for specific activities:
Pre-set flat is in
disp00. This generally is initialized once in
dmCtrlGUIand not touched in python.
The turbulence/aberration channel is
The AO channel is
disp03. Note that any AO correction needs to be multiplied by
-1to work properly.
Therefore, an example of initializing the BMC 1K DM’s turbulence and AO channels would be:
$ dm_turb = ImageStream('dm00disp11') $ dm_ao = ImageStream('dm00disp03')
From here, you can write the commands to the DM using the
<dm_com_mat> is the DM command matrix parameter to pass in. Note that
<dm_com_mat> will be different
based on the DM being used.
The BMC 1K is a
32x32matrix which assumes the units are in
microns. Try to avoid using the edge actuators.
The IrisAO segmented DM is a
37x3matrix where each row is a specific segment and the 1st column is piston (in
microns), the 2nd is tip, and the 3rd is tilt (each in
For every new DM command you want to enter, you can keep updating it with the
When you are done with the DM, close it similar to saving an image:
$ dm_turb.close() $ dm_ao.close()