A Beginner’s Guide to Using MagAOX
A Bit of Book-Keeping
To ssh on to rtc and icc from our local machine rather than the VM, we need to set up a proxy jump through aoc:
cd ~/.ssh
make or edit a file named config to include:
- Host aoc
HostName exao1.magao-x.org
- Host rtc
HostName rtc ProxyJump aoc
- Host icc
HostName icc ProxyJump aoc
- Host *
User <your-username>
With this set up, you should now be able to jump to rtc and icc just by typing:
ssh rtc or ssh icc in a terminal without logging in to the VM. This is especially useful for using scp to copy files to local.
Getting Started
First, open a terminal window (henceforth terminal #1), navigate to the VM, start it up, and start the tunnels:
cd dir/to/MagAOX/
vagrant up # if not done already
vagrant ssh
xctrl startup
Now, open up Terminator (or any terminal emmulator like Konsole in kde), and split in half such that it has an upper window and a lower window.
Log in to the VM in both windows and then:
top window:
cursesINDI
bottom window:
tail -f /tmp/cursesINDI_logs.txt
The cursesINDI_logs.txt file will print updates to let you know that the connection to the INDI server, and thus to the hardware, is active and working the way it should be.
Note: When using cursesINDI, you can type the name of the device you want to scroll to it faster. For target properties that are toggle-able, press “t” for toggle, then “y” to confirm.
If cursesINDI or getINDI fails with a connection error, we might need to run the system startups as the INDI server is probably down. But first, check the VM:
# Exit the VM via ctrl+d or typing logout
vagrant reload
vagrant ssh
xctrl startup
getINDI
If the problem persists after doing this the following steps are needed:
Check rtc:
ssh rtc
xsupify
cd
bash ./cacao_startup_woofer.sh
bash ./cacao_startup_tweeter.sh
fpsCTRL #to check if DMCOMB has started
xctrl startup
xctrl status #to check that processes have started
If everything is green, logout of the ssh
Now check icc:
ssh icc
xsupify
cd
bash ./cacao_startup_dmncpc.sh
xctrl startup
xctrl status #to check that processes have started
fpsCTRL #to check if DMCOMB has started
If everything looks good, logout of the ssh
Now check aoc:
ssh aoc
xsupify
cd
xctrl startup
xctrl status
All processes should be green, but if isAOC is red:
xctrl startup
If this doesn’t fix the process:
xctrl restart isAOC
Now we need to check isICC and isRTC to make sure the INDI servers are connected
ssh icc
xsupify
getINDI
If there is no response, ie:
[xsup@exao3] $ getINDI
No *.*.* from localhost:7624
Do:
xctrl restart isICC
getINDI
If still no response:
xctrl shutdown
xctrl startup
xctrl status
getINDI
Now repeat the same process on rtc until getINDI gives a response. With this done, everything should be good to go to continue on as normal
================================================================================
In terminal #1, type:
getINDI
If a bunch of information is printed, it means you are connected to the INDI server.
Now power up the Power GUI in terminal #1:
pwrGUI &
A window with a bunch of sliders will pop open.
Important: (the following sliders should all be on when pwrGUI comes up, and left on when shutting down!) NEVER TURN OFF:
Set up the milkzmqClient with everything we (may) need, again in terminal #1:
milkzmqClient -p 9000 localhost camwfs camwfs_dark camtip camtip_dark camlowfs camlowfs_dark camsci1 camsci1_dark camsci2 camsci2_dark dm00disp00 dm00disp dm00dispST dm01disp00 dm01disp dm01dispST dm02disp00 dm02disp dm02dispST &
Now the real time image viewers can be turned on. Let’s start with camtip, the camera viewing light picked off of the Pyramid WFS tip.
now can power on camtip using the slider in pwrGUI:usbdu0
Turn on the real time viewer for camtip by typing the command:
rtimv -c rtimv_camtip.conf &
================================================================================ Note for if camtip has an error:
if camtip-sw has an error appear in the INDI log that lead it to shutdown, the process needs to be restarted:
ssh icc
xsupify
xctrl status # to verify the process is dead
xctrl restart camtip-sw
================================================================================
The DMs
Note: dm00 is the woofer, dm01 is the tweeter, and dm02 is the ncpc
The safety check for turning on the DMs is the tweeter humidity. The easiest way to check the humidity is to open the real time viewers for the DMs, and check the RH value printed for dm01:
rtimv -c rtimv_dm00disp.conf &
rtimv -c rtimv_dm01disp.conf &
rtimv -c rtimv_dm02disp.conf &
Looking at the viewer for dm01, the upper right corner should have “RH: #.#%” printed. This is a readout of rhtweeter.humidity.current in cursesINDI.
This means it you can also use:
getINDI rhtweeter.*.*
Read the number next to humidity.current
Important: Anything higher than 18% for the current humidity, do NOT power the DMs on, and post about it on Slack.
If humidity.current < 18, it is safe to turn on the 3 DMs in pwrGUI:pdu1, and post in Slack that MagAOX is in use.
cd /opt/MagAOX/source/MagAOXgit pullmake guis_installAlthough we are checking the humidity of dm01, the tweeter, to ensure the safety powering it up, we also start the other two DMs first in part to get the ALPAO DMs over their creep.
dmCtrlGUI dmwoofer &
dmCtrlGUI dmtweeter &
dmCtrlGUI dmncpc &
Once all 3 GUIs are open, press the “set flat” button on all 3 GUIs
================================================================================ If one of the DMs isn’t responding when trying to flatten it, its process is likely down (or hanging). To fix:
Follow steps in the Handbook for: Software Startup
Specifically, for dmwoofer and dmtweeter:
ssh rtc
xsupify
cd
bash ./cacao_startup_woofer.sh
bash .cacao_startup_tweeter.sh
xctrl restart isRTC
xctrl restart <dmwoofer or dmtweeter> # until it reconnects
For dmncpc
ssh icc
xsupify
cd
bash ./cacao_startup_dmncpc.sh
xctrl restart isICC
tmux attach -t dmncpc
# ctrl+c, up-arrow, press enter until it reconnects
After this process, the DM in question should be reconnected to the INDI server and work in the GUI again.
================================================================================
Now the Tip/Tilt mirrors
Turn on the Pupil Alignment GUI:
pupilGuideGUI &
now in pwrGUI:pdu1, turn on ttmmod, ttmpupil, and in pwrGUI:pdu2, turn on stagecamlens.
Note: the ttmmod slider automatically goes halfway and stays for a bit, and then all the way to the right after some time for safety
Now in the Pupil Alignment GUI, press the “set” button for the Modulation & Centering and the Pupil Steering sections. The Camera Lens section largely takes care of itself.
Now power up camwfs using the pwrGUI:pdu1 camwfs and run:
rtimv -c rtimv_camwfs.conf &
open the camera GUI for camwfs:
cameraGUI camwfs &
================================================================================ Note for if this doesn’t go right:
If the rtimv viewer for camwfs appears all white, and the cameraGUI for it is blank, the process for camwfs has failed. To check this, open a terminal and:
ssh rtc
xsupify
tmux a -t camwfs
This should give a log. If it has an error in it that has stopped the funtion (e.g. “no serial response”), you need to restart the driver. This is most easily done by doing ctrl-c to stop the process, and then the up-arrow key to get the correct command. Press enter. The rtimv image should look right, and the cameraGUI should no longer be blank. Detach the tmux shell with ctrl-b + d, and then you can close the connection to rtc and the terminal.
================================================================================
In the camwfs cameraGUI, the Mode should be bin2 (use the “…” button and pulldown bar to select bin2). Cool down camwfs by editing the Detector Temp to -40 Set camwfs FPS to 200
Now turn on shwfs in pwrGUI:dcdu1 (shwfs stands for wfs shutter). “(off)” should disapper at the camwfs cameraGUI shutter.
Note: The shutter is open with the slider to the left, closed with the slider to the right.
Now repeat the process for the science cameras, camsci1 and camsci2:
Power up camsci1 and camsci2 using the pwrGUI sliders for them.
rtimv -c rtimv_camsci1.conf &
rtimv -c rtimv_camsci2.conf &
open the camera GUI for camwfs:
cameraGUI camsci1 &
cameraGUI camsci2 &
Open the shutters for both cameras by sliding the shutter slider in their respective cameraGUI window.
Finally, turn source on in pwrGUI:pdu0 using source slider.
Note: At this point, the DMs are flat, the TT mirros are set, the shutter is open: check if there is light in camtip and camwfs
Filter Wheels, Final Power ups, and Final Checks
IMPORTANT: The proper order for powering on the filter wheels fwscind and fwtelsim is to turn on the DC power first in pwrGUI before the USB power:
Slide the sliders for fwscind fwtelsim in pwrGUI:dcdu1.
Now all sliders in dcdu0 and dcdu1 categories.
Now can turn on everything in usbdu0 except for camacq.
REMINDER: fwscind and fwtelsim in usbdu0 must be done after the dcdu1 ones!
Turn on sliders in usbdu1 except for flipacq and flipeye
now in pdu3, turn on fliptip and tableair
now in pdu2, turn everything on
Note: stagerot and stagezaber take a couple minutes to home; the INDI log in the split window terminal bottom will print when they are ready. Wait for that message.
Now perform the final checks:
1. In cursesINDI (the top window of the split window terminal), go to fwtelsim we want to be in ND1, so go to ND1 filterName and toggle it on.
Double check that fliptip is “in” (also in cursesINDI)
Check that flipwfsf is also “in” (also in cursesINDI)
Make sure that stagepickoff is “in” (also in cursesINDI)
Now Align the System
Follow steps in the Handbook for: System Pupil Alignment
Pointers for going through this procedure:
Make sure to check the list at the top of the linked page vs. your curseINDI. It is important for getting light on camtip, camlowfs, and camwfs, at the correct light level.
If you need to open the camlowfs shutter (e.g. the rtimv viewer for camlowfs has “SHUT” written in it):
cameraGUI camlowfs &
rtimv -c rtimv_camlowfs.conf &
Just toggle the shutter slider as you did on camwfs.
In the real time viewer for camlowfs, press z to get the yellow box. Move it over the pupil image to set viewer scales appropriately.
4. set Rtest_03um on dmtweeter: using the pulldown at the bottom of the dmtweeter GUI, select ‘Rtest_0p3um’, then click the “set test” button.
Use the Pupil Steering section of the Pupil Alignment GUI to center the pupil using the diagonal arrows at the bottom middle. Make the the edge actuators evenly illuminated. If needed, you can switch to “ND2” on fwtelsim (to reduce saturation).
Press the “zero test” button on the dmtweeter control GUI when done.
Now align the WFS
You need camwfs and the Camera Lens section in the Pupil Alignment GUI to do this
In the camwfs viewer, press “t” to get the target quadrants to show
Using the Camera Lens section in Pupil Alignment GUI, use the arrows at the bottom to move pupils to roughly be even in the 4 quadrants in the real time viewer.
press “t” again to turn off the quadrant targets.
Now go to Modulation & Centering:
click the box for delta from mean
Use the arrows at the bottom to make the light in each pupil close to equal. you want to move from the largest positive toward the negative (mainly will be on the diagonal motions)
- When they are pretty even, modulate:
type 1000 in Frequency taret and hit enter
type 3 in Radius and hit enter
now click the modulate button (watch camtip real time viewer to see it working)
- Go back to camwfs viewer and press “r”, then “P” (shift+p) to fit circles to the pupils
if the Red circles overlap or look funky, you aren’t getting enough signal
go to the camera gui for camwfs, and edit the FPS to be slower (should be at 200Hz)
- Go back to Camera Lens in the Pupil Alignment GUI
try to match the red circles to the green ones
click the delta from set pt box, and look at the avg. try to get them to be ~0.1 or less
do “P” (shift+p) again in the viewer to turn off the fitting lines
The system is now aligned!
Now for Cacao-ing
To begin, set up rtc in the mode to have the lowest latency:
ssh rtc
cat /proc/cpuinfo | grep MHz | wc -l
If this is 72:
sudo /opt/MagAOX/source/MagAOX/script/rtc_cpuset
sudo /opt/MagAOX/source/MagAOX/script/rtc_procset
cat /proc/cpuinfo | grep MHz | wc -l
Note: Do not switch to xsup to do this, as you need sudo!
Open another terminal (henceforth terminal #2):
ssh rtc
xsupify
cd /opt/MagAOX/cacao/tweeter
./aolconf
Note: If you close this loop on accident, start again with ./aolconf -n. This doesn’t load the shared memory stuff, so it will connect faster. This is likely not dangerous to do, but only if you have run ./aolconf already.
Now, open another terminal (terminal #3) and ssh to rtc:
xsupify
cd
procCTRL
Now change font size (by zooming out or otherwise, depending on your default terminal) until all the text fits. Press X (shift+x) to quit, and then restart the program so it can all be seen (will help detect crashes in CACAO, and let you know when processes have completed)
procCTRL
Note: To reset procCTRL, press “R” (shift+r) on the keyboard
Get the dm01disp and camwfs real time viewers up.
In terminal #2, the left and right arrow keys move between “select” and “exit” on the bottom; up and down arrow keys move in the menu.
Scroll down (via down arrow presses) to Configure/link AO loop.
Go to the camwfs cameraGUI and make sure it is running at 1kHz, as it is probably not due to the alignment steps (click in Frame Rate box, type 1000, hit enter, box will be blue when it reaches it. Then restretch the camwfs viewer with “r”)
Note: The 1000 Hz here is actually whatever speed the loop is being run at. It must be the same as the Modulation & Centering modulation frequency! If you change the loop frequency (say to 2kHz), you must change the modulation frequncy. To do that, go to the Pupil Alignment GUI, Modulation & Centering, enter the desired frequency, 2000, in the Frequency target box, hit enter, 3 in the Radius target box, hit enter, then press the modulate button.
Return to the Cacao GUI, scroll down to Acquire WFS Dark and hit enter. This will take dark frames. “D” (shift+D) in the viewer for cameras will show with/out dark subtraction.
5. Scroll down in Cacao now to Measure Hardware Latency and hit enter. Hit enter again on 100. It will run. procCTRL will list “lat aol1_dmC aol1_wfsim”, when complete, the message will say “Loop exit <time_stamp>”. Scroll up in the GUI to read Hardware Latency and hope to see < 2 frames. If it’s near 2 frames or larger, message the PI / Slack.
Go back to the Pupil Alignment GUI and check the centering using Modulation & Centering only (no need for other two again!)
Adjust to try to get the median fluxes deltas close to 0 using the arrows and small movements
Protip: Have No. Avgs set to 200 still
Getting a Response Matrix:
Now, in the Cacao GUI terminal (terminal #2), scroll to START AUTO SYSTEM CALIBRATION (new modes)
- Press enter
procCTRL will say loop exit on far right for two processes (dmpokeC2b both times, one for hadamard modes and one for low order zernike modes) with STOPPED as the status. If it crashed, it will say so with a red box.
Scroll up in Cacao to the line with AUTOMATIC SYSTEM CALIBRATION and hit enter to refresh the GUI
To save the Response Matrix:
Open a new terminal:
ssh rtc
xsupify
cd /opt/MagAOX/cacao/tweeter
ls -l zrespM.fits
2. If it’s there we can copy it to local. Open a terminal without any ssh’ing:
scp rtc:/opt/MagAOX/cacao/tweeter/zrespM.fits /home/.../
- It is also useful (and pretty much necessary) to copy some other files from this location as well in order to do reconstructions. These include:
dmmask.fits
wfsmask.fits
wfsref0.fits
wfsdark/wfsdark_<date>.fits
Closing the Loop!
Now use the right arrow to select Top at bottom of cacao window and press enter. This will take you back to the main cacao menu
open a terminal to monitor when this process completes:
ssh rtc
xsupify
tmux ls # to look for aol1mkmodes
tmux attach -t aol1mkmodes
“dologext: command not found”ill will be printed. Press enter and if you get a command prompt, and that means the process is done
Now scroll to Load shared memory from conf and press enter
Open a terminal to monitor when this process completes:
ssh rtc
tmux atach -t aol1SMload
# wait for process to complete (getting a bash prompt)
Now that 4 new processes have been launched, we need to update the script that is allowing us to run with low latency:
ssh rtc
sudo /opt/MagAOX/source/MagAOX/script/rtc_procset
It should run without any errors or warnings.
Congrats! You have closed the loop on the WFS!
You can test this by opening up the DM modes GUI for dmwoofer and adding an aberration. You should immediately see the opposite shape appear in the viewer for the tweeter, and the camwfs output not change. You can open the modes GUI for dmncpc as well:
dmModeGUI wooferModes &
dmModeGUI ncpcModes &
Preparing camsci1 for images
- Set up to get light on camsci1
- Go to cursesINDI and make the following changes:fwfpm.filterName -> open1fwsci1.filterName -> zfwsciND -> ND2 or ND3 (to stop saturation)
- Setting up ROI for camsci1:
- Click the ROI button on camsci1 cameraGUIMouse over the central pixel of the PSF in the camsci1 viewer to get its coordinateSet the coordinate you found to the X Center and Y center boxes with that coordinateSet Width and Height to 32 each press enter (32 for speed, 64 can work too)Click checkClick setPress z in camsci1 and move the gold box to restretch the color scale.Close the ROI gui
- Now we want more speed on the camera by a bit:
- Click the box for Readout Spd and use the target pulldown to change to emccd_10MHzClick Vert. Shift Spd and use the target pulldown to change to 0_7usset exp time [sec.] to 0.001 to force fastest readout
You should get an FPS of ~156.1 on camsci1 after making those changes. Once you know this FPS, you need to update the AO loop to use it:
To stop the loop, scroll down to LOOP SETTING and set loop gain to 0. Then press enter on “stop loop” under LOOP CONTROL
- Update the frame rate of camwfs and the modulation frequency to match the camsci1 FPS:
change FPS on camwfs update modulation frequency in Pupil Alignment GUI
Now that you can see the light in our ROI on camsci1, you may want to clean up the PSF a little. To do this, run eye doctor.
See Eye-Doctor for more details
In terminal #1:
dmModesGUI ncpcModes & # if not opened earlier
Open a new terminal:
ssh icc
xsupify
cd
dm_eye_doctor 7624 ncpcModes camsci1 5 2...10 0.25
It will print when it is done, and you can watch in NCPC DM Modes GUI, DMdm02disp, and camsci1 viewers to see what it is doing, and if it is imroving the PSF quality.
To do stuff in Python
open a terminal:
ssh aoc -L 9999:localhost:9999navigate to localhost:9999 in a browser
Useful Python stuff to know to build around:
from magpyx.utils import ImageStream
# getting camera images
cam = ImageStream('camwfs')
image = cam.grab_latest() # simplest way to grab an image
cam.close() # when you're done
# commanding DMs
dm = ImageStream('dm01disp04')
cmd = np.zeros_like(dm.buffer)
cmd[20,23] = 0.1 # a poke
dm.write(cmd) # send the command
dm.close() # when finished
# Magic recipe for matching how Cacao does its reconstructions
# subtract the wfsdark from the measurement
# Multiply by wfsmask
# normalize by the sum [ e.g. var /= var.sum() ]
# Subtract wfsref0
- To save fits files written in python here to local:
- Copy the fits files writen to /home/xsup/magic_portalGo to BoxMagAOX/magic_portalDownload the filesGo back to the terminal ssh’d to aoccd to /home/xsup/magic_portalremove the files you added
Safe Shutdown!
- Shutting down cacao:
- set loop gain to 0scroll to STOP control loop, and press enterscroll to Loop processes ON, press to STOP, and press enter
The 4 processes for the loop should show as “stopped” in procCTRL, and the loop counter should no longer be increasing. Use Top to get back to the main cacao menu, or Exit to close aolconf.
IMPORTANT: First is warm up the 4 EMCCDs! These are camwfs, camlowfs, camsci1, and camsci2 (the camsci’s cool automatically, even if you don’t use them)
Close the shutter on camwfs in its cameraGUI (slide slider to the right)
Close the shutter on camlowfs in its cameraGUI
Close the shutter on camsci1 in its cameraGUI
Close the shutter on camsci2 in its cameraGUI
IMPORTANT! You must turn off fwtelsim and fwscind in usbdu0 next (must be done before the dc power ones!!)
If the temperature is 20C, the cameras can be slid off. 19C is okay, <19C is not. Check cursesINDI or camera GUIs for the current temps.
IMPORTANT: Before being done, double check that “instcool” is still powered on in pwrGUI (this is important as it keeps the CPUs and such from overheating)
Go to cursesINDI, and stop set_latency: sysMonRTC.set_latency, toggle off
Now close all the windows and post in Slack that MagAOX is off.
================================================================================ Note if things go wrong:
If sliders in pwrGUI stop sliding off, it is possible that fwtelsim in usbdu0 has rebooted ICC. If this happens, we need to go ensure that the processes are all running, and that the INDI server is connected.
open a terminal:
ssh icc
xsupify
xctrl status # everything should be dead if ICC rebooted
xctrl startup
xctrl status # everything should be green again
getINDI
getINDI on ICC returns nothingIn order to reconnect, run
xctrl restart isICC
Once this completes, cursesINDI and the INDI log should reconnect, and running getINDI on ICC should return stuff.
You should now notice all of the options back in pwrGUI. Continue to shut off where you left off (likely at fwtelsim or fwscind)
================================================================================