xctrl
The process control interface for MagAO-X apps.
The xctrl script is installed on all MagAO-X systems (including VMs
and the test bed) for startup, shutdown, quick status checks, and
inspection of MagAO-X apps.
To use xctrl, open a terminal window on (or SSH in to) the computer
you want to control. (You do not need to xsupify for xctrl.)
Then, commands take the form of xctrl VERB [PROCNAME] ... where the
VERBs are explained below. Most commands can be run without a
PROCNAME and will target all the configured processes by default.
The valid names for PROCNAME are listed in
/opt/MagAOX/config/proclist_${MAGAOX_ROLE}.txt.
xctrl startup
When called without additional arguments, xctrl startup will start
any process from the proclist that does not have a running tmux
session. (Yes, each process is parented to a tmux session.)
If you list one or more process names,
e.g. xctrl startup camtip camsci, then xctrl will only consider
the processes named.
xctrl shutdown
When called without additional arguments, xctrl shutdown will
interrupt all running MagAO-X processes (by sending Ctrl+C to their
tmux sessions). If they do not cleanly exit after five attempts (~5
seconds), xctrl will terminate them.
You can list one or more process names, similarly to startup.
xctrl restart
Effectively xctrl shutdown followed by xctrl startup.
You can list one or more process names, similarly to startup.
xctrl inspect PROCNAME
Requires an argument: the process name to inspect. The xctrl inspect
command attaches to the tmux session where PROCNAME is running
(or, if you’re inspecting it, maybe recently crashed). From the prompt
you can use the up arrow key as normal to retrieve the last command
executed (i.e. the process under inspection).
When you’re done inspecting, you can use Ctrl+b followed by d to
detach from a running process (or just exit to end the session).
xctrl status
Displays a short status report for all configured processes. If you supply one or more PROCNAMEs, it will show only those. Example output:
$ xctrl status
isAOC: running (pid: 60636)
pdu0: running (pid: 6181)
pdu1: running (pid: 16067)
pdu2: running (pid: 21664)
pdu3: running (pid: 11874)
labCool: running (pid: 11956)
tcsi: running (pid: 11843)
adctrack: running (pid: 12061)
aoc_icc_indi: running (pid: 12057)
aoc_rtc_indi: running (pid: 12054)
aoc_icc_milkzmq: running (pid: 11883)
aoc_rtc_milkzmq: running (pid: 12150)
mzmqClientAOC_ICC: running (pid: 12149)
mzmqClientAOC_RTC: running (pid: 12045)
The statuses will be color-coded (if your terminal supports it). Green means it’s all good.
You can also check the status of a single process with
xctrl status PROCNAME, e.g.:
$ xctrl status isAOC
isAOC: running (pid: 60636)
If you shut down a process cleanly (or haven’t yet started it) you will
see not started in yellow and (usually) log lines indicating the
state transitioned to SHUTDOWN:
isAOC: not started
If the process dies but its parent tmux session is alive:
isAOC: session exists, but process is not running
You may want to investigate the recent logs with logdump
or xctrl peek PROCNAME to see why it died. To restore it, you can
xctrl inspect PROCNAME to connect to that tmux session and
attempt to restart the process. You can also xctrl restart PROCNAME
to end and recreate the tmux session.
It is possible on occasion for a process to die, e.g. after a system crash and reboot. In those cases, you will see something like this:
isAOC: dead (stale pid)
This is another situation where you may want to check recent logs. To
restore it, you can simply xctrl startup that process.
xctrl peek
The extended version of xctrl status, which includes both the status
and last ten lines that the process wrote to its log. (For a full log
dump, see logdump docs.) You can also use
xctrl peek PROCNAME to peek at a single process, e.g.:
$ xctrl peek isAOC
isAOC: running (pid: 60636)
2020-07-30T16:10:37.150000000 INFO IS: Driver camwfs-dark@localhost:7626 at 127.0.0.1 now connected on socket=91
2020-07-30T16:10:37.150000000 INFO IS: Driver dmtweeter@localhost:7626 at 127.0.0.1 now connected on socket=92
2020-07-30T16:10:37.178000000 INFO IS: Driver ttmpupil@localhost:7626 at 127.0.0.1 now connected on socket=93
2020-07-30T16:10:37.178000000 INFO IS: Driver camwfs-slopes@localhost:7626 at 127.0.0.1 now connected on socket=94
2020-07-30T16:10:37.211000000 INFO IS: Driver fxngenmodwfs@localhost:7626 at 127.0.0.1 now connected on socket=95
2020-07-30T16:10:37.218000000 INFO IS: Driver aoloop@localhost:7626 at 127.0.0.1 now connected on socket=96
2020-07-30T16:10:37.219000000 INFO IS: Driver tweeterModes@localhost:7626 at 127.0.0.1 now connected on socket=97
2020-07-30T16:10:37.219000000 INFO IS: Driver wooferModes@localhost:7626 at 127.0.0.1 now connected on socket=98
2020-07-30T16:10:37.220000000 INFO IS: Driver w2tcsOffloader@localhost:7626 at 127.0.0.1 now connected on socket=99
2020-07-30T16:10:37.259000000 INFO IS: Driver dmtweeter-avg@localhost:7626 at 127.0.0.1 now connected on socket=100
End isAOC logs
Tip: Lines like State changed from A to B can give you a hint as
to what’s going wrong. See
stateCodes.hpp
for descriptions of the different states.