.. raw:: html
.. role:: red
.. raw:: html
.. role:: blue
.. raw:: html
.. role:: green
.. raw:: html
.. role:: orange
.. raw:: html
.. role:: gray
**********************************
A Beginner's Guide to Using MagAOX
**********************************
A Bit of Book-Keeping
=====================
To ssh on to rtc and icc from our local machine rather than the VM, we need to set up a proxy jump through aoc:
.. code-block:: bash
cd ~/.ssh
make or edit a file named config to include:
Host aoc
HostName exao1.magao-x.org
Host rtc
HostName rtc
ProxyJump aoc
Host icc
HostName icc
ProxyJump aoc
Host *
User
With this set up, you should now be able to jump to rtc and icc just by typing:
``ssh rtc`` or ``ssh icc`` in a terminal without logging in to the VM. This is especially useful for using scp to copy files to local.
Getting Started
===============
First, open a terminal window (henceforth terminal #1), navigate to the VM, start it up, and start the tunnels:
.. code-block:: bash
cd dir/to/MagAOX/
vagrant up # if not done already
vagrant ssh
xctrl startup
Now, open up Terminator (or any terminal emmulator like Konsole in kde), and split in half such that it has an upper window and a lower window.
Log in to the VM in both windows and then:
top window:
.. code-block:: bash
cursesINDI
bottom window:
.. code-block:: bash
tail -f /tmp/cursesINDI_logs.txt
The cursesINDI_logs.txt file will print updates to let you know that the connection to the INDI server, and thus to the hardware, is active and working the way it should be.
:blue:`Note`:
When using cursesINDI, you can type the name of the device you want to scroll to it faster. For target properties that are toggle-able, press "t" for toggle, then "y" to confirm.
| :green:`================================================================================`
| :green:`In case this goes wrong`:
If cursesINDI or getINDI fails with a connection error, we might need to run the system startups as the INDI server is probably down. But first, check the VM:
.. code-block:: bash
# Exit the VM via ctrl+d or typing logout
vagrant reload
vagrant ssh
xctrl startup
getINDI
If the problem persists after doing this the following steps are needed:
Check rtc:
.. code-block:: bash
ssh rtc
xsupify
cd
bash ./cacao_startup_woofer.sh
bash ./cacao_startup_tweeter.sh
fpsCTRL #to check if DMCOMB has started
xctrl startup
xctrl status #to check that processes have started
If everything is green, logout of the ssh
Now check icc:
.. code-block:: bash
ssh icc
xsupify
cd
bash ./cacao_startup_dmncpc.sh
xctrl startup
xctrl status #to check that processes have started
fpsCTRL #to check if DMCOMB has started
If everything looks good, logout of the ssh
Now check aoc:
.. code-block:: bash
ssh aoc
xsupify
cd
xctrl startup
xctrl status
All processes should be green, but if isAOC is red:
``xctrl startup``
If this doesn't fix the process:
``xctrl restart isAOC``
Now we need to check isICC and isRTC to make sure the INDI servers are connected
.. code-block:: bash
ssh icc
xsupify
getINDI
If there is no response, ie:
.. code-block:: bash
[xsup@exao3] $ getINDI
No *.*.* from localhost:7624
Do:
.. code-block:: bash
xctrl restart isICC
getINDI
If still no response:
.. code-block:: bash
xctrl shutdown
xctrl startup
xctrl status
getINDI
Now repeat the same process on rtc until getINDI gives a response. With this done, everything should be good to go to continue on as normal
:green:`================================================================================`
In terminal #1, type:
.. code-block:: bash
getINDI
If a bunch of information is printed, it means you are connected to the INDI server.
Now power up the Power GUI in terminal #1:
.. code-block:: bash
pwrGUI &
A window with a bunch of sliders will pop open.
:red:`Important`:
(the following sliders should all be on when pwrGUI comes up, and left on when shutting down!)
**NEVER TURN OFF**:
| :orange:`pdu0`:
| compicc
| comprtc
| dcpwr
| swinst
| :orange:`pdu3`:
| blower
| fan1
| fan2
| instcool
Set up the milkzmqClient with everything we (may) need, again in terminal #1:
.. code-block:: bash
milkzmqClient -p 9000 localhost camwfs camwfs_dark camtip camtip_dark camlowfs camlowfs_dark camsci1 camsci1_dark camsci2 camsci2_dark dm00disp00 dm00disp dm00dispST dm01disp00 dm01disp dm01dispST dm02disp00 dm02disp dm02dispST &
Now the real time image viewers can be turned on. Let's start with camtip, the camera viewing light picked off of the Pyramid WFS tip.
1. now can power on camtip using the slider in :orange:`pwrGUI:usbdu0`
2. Turn on the real time viewer for camtip by typing the command:
.. code-block:: bash
rtimv -c rtimv_camtip.conf &
:green:`================================================================================`
:green:`Note for if camtip has an error`:
if camtip-sw has an error appear in the INDI log that lead it to shutdown, the process needs to be restarted:
.. code-block:: bash
ssh icc
xsupify
xctrl status # to verify the process is dead
xctrl restart camtip-sw
:green:`================================================================================`
The DMs
=======
:blue:`Note`:
dm00 is the woofer, dm01 is the tweeter, and dm02 is the ncpc
The safety check for turning on the DMs is the tweeter humidity. The easiest way to check the humidity is to open the real time viewers for the DMs, and check the RH value printed for dm01:
.. code-block:: bash
rtimv -c rtimv_dm00disp.conf &
rtimv -c rtimv_dm01disp.conf &
rtimv -c rtimv_dm02disp.conf &
Looking at the viewer for dm01, the upper right corner should have "RH: #.#%" printed. This is a readout of rhtweeter.humidity.current in cursesINDI.
This means it you can also use:
.. code-block:: bash
getINDI rhtweeter.*.*
Read the number next to humidity.current
:red:`Important`:
Anything higher than **18%** for the current humidity, **do NOT** power the DMs on, and post about it on Slack.
If humidity.current < 18, it is safe to turn on the 3 DMs in :orange:`pwrGUI:pdu1`, and post in Slack that MagAOX is in use.
| If the real time viewer for dm01 does not have the humidity printed, the cameraGUI isn't building:
| 1. post in Slack software channel, then when notified of fix:
| 2. ``cd /opt/MagAOX/source/MagAOX``
| 3. ``git pull``
| 4. ``make guis_install``
| This should fix the problem.
Although we are checking the humidity of dm01, the tweeter, to ensure the safety powering it up, we also start the other two DMs first in part to get the ALPAO DMs over their creep.
.. code-block:: bash
dmCtrlGUI dmwoofer &
dmCtrlGUI dmtweeter &
dmCtrlGUI dmncpc &
Once all 3 GUIs are open, press the "set flat" button on all 3 GUIs
:green:`================================================================================`
If one of the DMs isn't responding when trying to flatten it, its process is likely down (or hanging). To fix:
**Follow steps in the Handbook for:**
:doc:`Software Startup <./startup>`
Specifically, for dmwoofer and dmtweeter:
.. code-block:: bash
ssh rtc
xsupify
cd
bash ./cacao_startup_woofer.sh
bash .cacao_startup_tweeter.sh
xctrl restart isRTC
xctrl restart # until it reconnects
For dmncpc
.. code-block:: bash
ssh icc
xsupify
cd
bash ./cacao_startup_dmncpc.sh
xctrl restart isICC
tmux attach -t dmncpc
# ctrl+c, up-arrow, press enter until it reconnects
After this process, the DM in question should be reconnected to the INDI server and work in the GUI again.
:green:`================================================================================`
| :blue:`Note`:
| "[" on the keyboard to square rtimv viewers as big as possible (scaled to the data)
| "]" makes square as small as possible
Now the Tip/Tilt mirrors
========================
Turn on the Pupil Alignment GUI:
.. code-block:: bash
pupilGuideGUI &
now in :orange:`pwrGUI:pdu1`, turn on ttmmod, ttmpupil, and in :orange:`pwrGUI:pdu2`, turn on stagecamlens.
:blue:`Note`: the ttmmod slider automatically goes halfway and stays for a bit, and then all the way to the right after some time for safety
Now in the Pupil Alignment GUI, press the "set" button for the Modulation & Centering and the Pupil Steering sections. The Camera Lens section largely takes care of itself.
Now power up camwfs using the :orange:`pwrGUI:pdu1` camwfs and run:
.. code-block:: bash
rtimv -c rtimv_camwfs.conf &
open the camera GUI for camwfs:
.. code-block:: bash
cameraGUI camwfs &
:green:`================================================================================`
:green:`Note for if this doesn't go right`:
If the rtimv viewer for camwfs appears all white, and the cameraGUI for it is blank, the process for camwfs has failed. To check this, open a terminal and:
.. code-block:: bash
ssh rtc
xsupify
tmux a -t camwfs
This should give a log. If it has an error in it that has stopped the funtion (e.g. "no serial response"), you need to restart the driver. This is most easily done by doing **ctrl-c** to stop the process, and then the up-arrow key to get the correct command. Press enter. The rtimv image should look right, and the cameraGUI should no longer be blank. Detach the tmux shell with **ctrl-b + d**, and then you can close the connection to rtc and the terminal.
:green:`================================================================================`
In the camwfs cameraGUI, the Mode should be bin2 (use the "..." button and pulldown bar to select bin2).
Cool down **camwfs** by editing the **Detector Temp** to **-40**
Set **camwfs** FPS to **200**
Now turn on shwfs in :orange:`pwrGUI:dcdu1` (shwfs stands for wfs shutter).
"(off)" should disapper at the camwfs cameraGUI shutter.
:blue:`Note`: The shutter is open with the slider to the left, closed with the slider to the right.
Now repeat the process for the science cameras, camsci1 and camsci2:
Power up camsci1 and camsci2 using the pwrGUI sliders for them.
.. code-block:: bash
rtimv -c rtimv_camsci1.conf &
rtimv -c rtimv_camsci2.conf &
open the camera GUI for camwfs:
.. code-block:: bash
cameraGUI camsci1 &
cameraGUI camsci2 &
Open the shutters for both cameras by sliding the shutter slider in their respective cameraGUI window.
Finally, turn source on in :orange:`pwrGUI:pdu0` using source slider.
:blue:`Note`: At this point, the DMs are flat, the TT mirros are set, the shutter is open:
check if there is light in camtip and camwfs
Filter Wheels, Final Power ups, and Final Checks
================================================
:red:`IMPORTANT`: The proper order for powering on the filter wheels fwscind and fwtelsim is to turn on the DC power first in pwrGUI before the USB power:
1. Slide the sliders for fwscind fwtelsim in :orange:`pwrGUI:dcdu1`.
2. Now all sliders in :orange:`dcdu0` and :orange:`dcdu1` categories.
3. Now can turn on everything in :orange:`usbdu0` except for camacq.
:red:`REMINDER`: fwscind and fwtelsim in :orange:`usbdu0` must be done after the dcdu1 ones!
4. Turn on sliders in :orange:`usbdu1` except for flipacq and flipeye
5. now in :orange:`pdu3`, turn on fliptip and tableair
6. now in :orange:`pdu2`, turn everything on
:blue:`Note`: stagerot and stagezaber take a couple minutes to home; the INDI log in the split window terminal bottom will print when they are ready. Wait for that message.
| :blue:`Note`: Now in pwrGUI, everything is on except:
| evncontweeter and flipacq (in :orange:`pdu3`)
| camacq and flipacq (in :orange:`usbdu1`)
| flipeye
Now perform the final checks:
1. In cursesINDI (the top window of the split window terminal), go to fwtelsim
we want to be in ND1, so go to ND1 filterName and toggle it on.
2. Double check that fliptip is "in" (also in cursesINDI)
3. Check that flipwfsf is also "in" (also in cursesINDI)
4. Make sure that stagepickoff is "in" (also in cursesINDI)
Now Align the System
====================
**Follow steps in the Handbook for:**
:doc:`System Pupil Alignment <./alignment>`
:green:`Pointers for going through this procedure`:
1. Make sure to check the list at the top of the linked page vs. your curseINDI. It is **important** for getting light on camtip, camlowfs, and camwfs, at the correct light level.
2. If you need to open the camlowfs shutter (e.g. the rtimv viewer for camlowfs has "SHUT" written in it):
.. code-block:: bash
cameraGUI camlowfs &
rtimv -c rtimv_camlowfs.conf &
Just toggle the shutter slider as you did on camwfs.
3. In the real time viewer for camlowfs, press z to get the yellow box. Move it over the pupil image to set viewer scales appropriately.
4. set Rtest_03um on dmtweeter:
using the pulldown at the bottom of the dmtweeter GUI, select 'Rtest_0p3um', then click the "set test" button.
5. Use the Pupil Steering section of the Pupil Alignment GUI to center the pupil using the diagonal arrows at the bottom middle. Make the the edge actuators evenly illuminated. If needed, you can switch to "ND2" on fwtelsim (to reduce saturation).
6. Press the "zero test" button on the dmtweeter control GUI when done.
Now align the WFS
=================
You need camwfs and the Camera Lens section in the Pupil Alignment GUI to do this
1. In the camwfs viewer, press "t" to get the target quadrants to show
2. Using the **Camera Lens** section in Pupil Alignment GUI, use the arrows at the bottom to move pupils to roughly be even in the 4 quadrants in the real time viewer.
3. press "t" again to turn off the quadrant targets.
Now go to **Modulation & Centering**:
1. click the box for delta from mean
2. Use the arrows at the bottom to make the light in each pupil close to equal. you want to move from the largest positive toward the negative (mainly will be on the diagonal motions)
3. When they are pretty even, modulate:
- type 1000 in Frequency taret and hit enter
- type 3 in Radius and hit enter
- now click the modulate button (watch camtip real time viewer to see it working)
4. Go back to camwfs viewer and press "r", then "P" (shift+p) to fit circles to the pupils
- if the Red circles overlap or look funky, you aren't getting enough signal
- go to the camera gui for camwfs, and edit the FPS to be slower (should be at 200Hz)
5. Go back to **Camera Lens** in the Pupil Alignment GUI
- try to match the red circles to the green ones
- click the delta from set pt box, and look at the avg. try to get them to be ~0.1 or less
- do "P" (shift+p) again in the viewer to turn off the fitting lines
:blue:`The system is now aligned!`
Now for Cacao-ing
=================
To begin, set up rtc in the mode to have the lowest latency:
.. code-block:: bash
ssh rtc
cat /proc/cpuinfo | grep MHz | wc -l
If this is 72:
.. code-block:: bash
sudo /opt/MagAOX/source/MagAOX/script/rtc_cpuset
sudo /opt/MagAOX/source/MagAOX/script/rtc_procset
cat /proc/cpuinfo | grep MHz | wc -l
:blue:`Note`: Do not switch to xsup to do this, as you need sudo!
| It should now be 54. Now we are ready to set rtc into lowest latency mode:
| go to cursesINDI
| search sysMonRTC
| go to sysMonRTC.set_latency and toggle it on
Open another terminal (henceforth terminal #2):
.. code-block:: bash
ssh rtc
xsupify
cd /opt/MagAOX/cacao/tweeter
./aolconf
:blue:`Note`: If you close this loop on accident, start again with ``./aolconf -n``. This doesn't load the shared memory stuff, so it will connect faster. This is likely not dangerous to do, but only if you have run ``./aolconf`` already.
Now, open another terminal (terminal #3) and ssh to rtc:
.. code-block:: bash
xsupify
cd
procCTRL
Now change font size (by zooming out or otherwise, depending on your default terminal) until all the text fits.
Press X (shift+x) to quit, and then restart the program so it can all be seen (will help detect crashes in CACAO, and let you know when processes have completed)
.. code-block:: bash
procCTRL
:blue:`Note`: To reset procCTRL, press "R" (shift+r) on the keyboard
1. Get the dm01disp and camwfs real time viewers up.
In terminal #2, the left and right arrow keys move between "select" and "exit" on the bottom; up and down arrow keys move in the menu.
2. Scroll down (via down arrow presses) to **Configure/link AO loop**.
3. Go to the **camwfs cameraGUI** and make sure it is running at 1kHz, as it is probably not due to the alignment steps (click in Frame Rate box, type 1000, hit enter, box will be blue when it reaches it. Then restretch the camwfs viewer with "r")
:blue:`Note`: The 1000 Hz here is actually whatever speed the loop is being run at. It must be the same as the **Modulation & Centering** modulation frequency! If you change the loop frequency (say to 2kHz), you must change the modulation frequncy. To do that, go to the **Pupil Alignment GUI**, **Modulation & Centering**, enter the desired frequency, 2000, in the Frequency target box, hit enter, 3 in the Radius target box, hit enter, then press the modulate button.
4. Return to the Cacao GUI, scroll down to **Acquire WFS Dark** and hit enter. This will take dark frames. "D" (shift+D) in the viewer for cameras will show with/out dark subtraction.
5. Scroll down in Cacao now to **Measure Hardware Latency** and hit enter. Hit enter again on 100. It will run. procCTRL will list "lat aol1_dmC aol1_wfsim", when complete, the message will say "Loop exit ".
Scroll up in the GUI to read **Hardware Latency** and hope to see < 2 frames.
If it's near 2 frames or larger, message the PI / Slack.
6. Go back to the Pupil Alignment GUI and check the centering using **Modulation & Centering** only (no need for other two again!)
7. Adjust to try to get the median fluxes deltas close to 0 using the arrows and small movements
:green:`Protip`: Have No. Avgs set to 200 still
Getting a Response Matrix:
==========================
1. Now, in the Cacao GUI terminal (terminal #2), scroll to **START AUTO SYSTEM CALIBRATION (new modes)**
2. Press enter
- procCTRL will say loop exit on far right for two processes (dmpokeC2b both times, one for hadamard modes and one for low order zernike modes) with STOPPED as the status. If it crashed, it will say so with a red box.
3. Scroll up in Cacao to the line with **AUTOMATIC SYSTEM CALIBRATION** and hit enter to refresh the GUI
To save the Response Matrix:
1. Open a new terminal:
.. code-block:: bash
ssh rtc
xsupify
cd /opt/MagAOX/cacao/tweeter
ls -l zrespM.fits
2. If it's there we can copy it to local. Open a terminal without any ssh'ing:
``scp rtc:/opt/MagAOX/cacao/tweeter/zrespM.fits /home/.../``
3. It is also useful (and pretty much necessary) to copy some other files from this location as well in order to do reconstructions. These include:
- dmmask.fits
- wfsmask.fits
- wfsref0.fits
- wfsdark/wfsdark_.fits
Closing the Loop!
=================
1. Now use the right arrow to select :gray:`Top` at bottom of cacao window and press enter. This will take you back to the main cacao menu
| 2. Select: **Modes and Control Matrix** and press enter
| scroll to:
| modeCPA, press enter, and change to 24
| mkModes0, press enter
3. open a terminal to monitor when this process completes:
.. code-block:: bash
ssh rtc
xsupify
tmux ls # to look for aol1mkmodes
tmux attach -t aol1mkmodes
"dologext: command not found"ill will be printed. Press enter and if you get a command prompt, and that means the process is done
| 4. Go back to cacao window:
| Take note of the date in **STAGED CONFIGURATION**: press enter and verify the date changes
| Arrow down and press enter on **Configuration Update**, check **ACTIVE CONFIGURATION** below to see its date matches the new one in **Staged configuration**.
5. Now scroll to **Load shared memory from conf** and press enter
6. Open a terminal to monitor when this process completes:
.. code-block:: bash
ssh rtc
tmux atach -t aol1SMload
# wait for process to complete (getting a bash prompt)
| 7. now select :gray:`Top` again in the cacao window
| Select **Control AO (L)oop**
| 8. scroll down to:
| **Loop processes ON, press to STOP** and press enter
| 9. scroll down to:
| **START loop processes** and press enter
| check procCTRL; you should see 4 new processes that all stay green (Active)
10. Now that 4 new processes have been launched, we need to update the script that is allowing us to run with low latency:
.. code-block:: bash
ssh rtc
sudo /opt/MagAOX/source/MagAOX/script/rtc_procset
It should run without any errors or warnings.
| 11. Scroll down to **LOOP SETTING**
| Verify loop gain is 0
| 12. Scroll to **LOOP CONTROL**
| Highlight **START control loop** and press enter
| Check procCTRL again, and see that loopcnt is updating at the frame rate
| 13. Go to loop gain under **LOOP SETTING** again:
| Increase the value little by little and verify the loop is stable
It is stable if dm01disp (the tweeter) has no saturated actuators, and is not oscillating between values
Congrats! You have closed the loop on the WFS!
You can test this by opening up the DM modes GUI for dmwoofer and adding an aberration. You should immediately see the opposite shape appear in the viewer for the tweeter, and the camwfs output not change. You can open the modes GUI for dmncpc as well:
.. code-block:: bash
dmModeGUI wooferModes &
dmModeGUI ncpcModes &
Preparing camsci1 for images
============================
Set up to get light on camsci1
| Go to cursesINDI and make the following changes:
| fwfpm.filterName -> open1
| fwsci1.filterName -> z
| fwsciND -> ND2 or ND3 (to stop saturation)
Setting up ROI for camsci1:
| Click the ROI button on camsci1 cameraGUI
| Mouse over the central pixel of the PSF in the camsci1 viewer to get its coordinate
| Set the coordinate you found to the X Center and Y center boxes with that coordinate
| Set Width and Height to 32 each press enter (32 for speed, 64 can work too)
| Click check
| Click set
| Press z in camsci1 and move the gold box to restretch the color scale.
| Close the ROI gui
Now we want more speed on the camera by a bit:
| Click the box for Readout Spd and use the target pulldown to change to emccd_10MHz
| Click Vert. Shift Spd and use the target pulldown to change to 0_7us
| set exp time [sec.] to 0.001 to force fastest readout
You should get an FPS of ~156.1 on camsci1 after making those changes. Once you know this FPS, you need to update the AO loop to use it:
1. To stop the loop, scroll down to **LOOP SETTING** and set loop gain to 0. Then press enter on "stop loop" under **LOOP CONTROL**
2. Update the frame rate of camwfs and the modulation frequency to match the camsci1 FPS:
change FPS on camwfs
update modulation frequency in Pupil Alignment GUI
Now that you can see the light in our ROI on camsci1, you may want to clean up the PSF a little. To do this, run eye doctor.
1. See :doc:`Eye-Doctor <./software/utils/eyedoctor>` for more details
In terminal #1:
.. code-block:: bash
dmModesGUI ncpcModes & # if not opened earlier
Open a new terminal:
.. code-block:: bash
ssh icc
xsupify
cd
dm_eye_doctor 7624 ncpcModes camsci1 5 2...10 0.25
It will print when it is done, and you can watch in NCPC DM Modes GUI, DMdm02disp, and camsci1 viewers to see what it is doing, and if it is imroving the PSF quality.
To do stuff in Python
=====================
1. open a terminal:
2. ``ssh aoc -L 9999:localhost:9999``
3. navigate to localhost:9999 in a browser
Useful Python stuff to know to build around:
.. code-block:: python
from magpyx.utils import ImageStream
# getting camera images
cam = ImageStream('camwfs')
image = cam.grab_latest() # simplest way to grab an image
cam.close() # when you're done
# commanding DMs
dm = ImageStream('dm01disp04')
cmd = np.zeros_like(dm.buffer)
cmd[20,23] = 0.1 # a poke
dm.write(cmd) # send the command
dm.close() # when finished
# Magic recipe for matching how Cacao does its reconstructions
# subtract the wfsdark from the measurement
# Multiply by wfsmask
# normalize by the sum [ e.g. var /= var.sum() ]
# Subtract wfsref0
To save fits files written in python here to local:
| Copy the fits files writen to /home/xsup/magic_portal
| Go to Box
| MagAOX/magic_portal
| Download the files
| Go back to the terminal ssh'd to aoc
| cd to /home/xsup/magic_portal
| remove the files you added
Safe Shutdown!
==============
Shutting down cacao:
| set loop gain to 0
| scroll to STOP control loop, and press enter
| scroll to Loop processes ON, press to STOP, and press enter
The 4 processes for the loop should show as "stopped" in procCTRL, and the loop counter should no longer be increasing. Use :gray:`Top` to get back to the main cacao menu, or :gray:`Exit` to close aolconf.
:red:`IMPORTANT`: First is warm up the 4 EMCCDs! These are camwfs, camlowfs, camsci1, and camsci2 (the camsci's cool automatically, even if you don't use them)
| 1. Edit in their camera GUI temp box, or through cursesINDI:
| In cursesINDI, go to the device and edit the .temp_ccd target to 20.
| For example:
| type "camsci1" in cursesINDI, go to target for temp_ccd, press "e", type 20, then press enter, then press "y"
2. Close the shutter on camwfs in its cameraGUI (slide slider to the right)
3. Close the shutter on camlowfs in its cameraGUI
4. Close the shutter on camsci1 in its cameraGUI
5. Close the shutter on camsci2 in its cameraGUI
| 5. **For the 3 DMs and 2 Tip/Tilt mirrors:**
| For DMs, press the "zero flat" button and then "release" button
| It is now safe to slide the DMs off in :orange:`pwrGUI` (dmtweeter, dmwoofer, and dmncpc)
| 6. **For ttmmod and ttmpupil:**
| go to the Pupil Alignment GUI and hit the "set" button under **Modulation & Centering**
| once it is in SET state, it is safe to hit "rest" for **Modulation & Centering** and **Pupil Steering**
| Both will have state set to RIP
| It is now safe to slide off ttmmod and ttmpupil in :orange:`pwrGUI`
:red:`IMPORTANT!` You must turn off fwtelsim and fwscind in :orange:`usbdu0` next (must be done before the dc power ones!!)
| 7. Now everything in pwrGUI except the following can be off:
| :orange:`cameras`:
| camwfs
| camlowfs
| camsci1
| camsci2
| :orange:`pdu0`:
| compicc
| comprtc
| dcpwr
| swinst
| :orange:`pdu3`:
| blower
| fan1
| fan2
| instcool
If the temperature is 20C, the cameras can be slid off. 19C is okay, <19C is not. Check cursesINDI or camera GUIs for the current temps.
:red:`IMPORTANT`: Before being done, double check that "instcool" is still powered on in pwrGUI (this is important as it keeps the CPUs and such from overheating)
Go to cursesINDI, and stop set_latency:
sysMonRTC.set_latency, toggle off
Now close all the windows and post in Slack that MagAOX is off.
:green:`================================================================================`
:green:`Note if things go wrong:`
If sliders in pwrGUI stop sliding off, it is possible that fwtelsim in usbdu0 has rebooted ICC. If this happens, we need to go ensure that the processes are all running, and that the INDI server is connected.
open a terminal:
.. code-block:: bash
ssh icc
xsupify
xctrl status # everything should be dead if ICC rebooted
xctrl startup
xctrl status # everything should be green again
getINDI
| getINDI probably still won't connect. Symptoms of this include:
| 1. ``getINDI`` on ICC returns nothing
| 2. Inspecting the log of one of the processes (e.g. logdump -f camlowfs) will show "waiting for power state"
| 3. pwrGUI from the VM or AOC will only have pdu0-3 (and nothing else)
In order to reconnect, run
.. code-block:: bash
xctrl restart isICC
Once this completes, cursesINDI and the INDI log should reconnect, and running ``getINDI`` on ICC should return stuff.
You should now notice all of the options back in pwrGUI. Continue to shut off where you left off (likely at fwtelsim or fwscind)
:green:`================================================================================`