Networking ========== This document uses the hostnames of the machines interchangeably with their roles. For reference: - ``exao1`` — AOC - ``exao2`` — RTC - ``exao3`` — ICC Topology -------- **University of Arizona** .. figure:: ua_network.svg :alt: Schematic representation of the connections between our lab computers when at home at University of Arizona Schematic representation of the connections between our lab computers when at home at University of Arizona **Las Campanas Observatory** TODO Ubuntu and NetworkManager and ufw and firewalld ----------------------------------------------- Ubuntu does not use NetworkManager, so you have to change to NetworkManager. This is done in `switch_ubuntu_networkmanager.sh `_ when setting up the system, and again as updates break it. The firewall of choice in Ubuntu is UFW, but firewalld integrates with NetworkManager so we can tag connections as trusted rather than interfaces. Install:: sudo apt install firewalld sudo systemctl enable firewalld sudo systemctl start firewalld ufw disable Firewall zones -------------- Certain interfaces are instrument internal: rack LAN, cameras, and direct NIC-to-NIC links. To ensure traffic is unrestricted on them, configure as follows: - ``exao1``, ``exao2``, and ``exao3`` - ``sudo nmcli con modify instrument connection.zone trusted`` - ``exao2`` only - ``sudo nmcli con modify rtc-to-icc connection.zone trusted`` - ``exao3`` only - ``sudo nmcli con modify icc-to-rtc connection.zone trusted`` - ``sudo nmcli con modify camsci1 connection.zone trusted`` - ``sudo nmcli con modify camsci2 connection.zone trusted`` Network Connections ------------------- exao1 ~~~~~ +------------------+------------------+------------------+----------------+--------------------------+----------------------+-----------------+ | connection name | device | IPv4 address | subnet mask | default route / gateway | DNS servers | search domains | +==================+==================+==================+================+==========================+======================+=================+ | instrument | enx2cfda1c61dde | 192.168.0.10 | 255.255.255.0 | 192.168.0.1 | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+----------------------+-----------------+ | www-ua | enx2cfda1c61ddf | (DHCP) | +------------------+------------------+------------------+----------------+--------------------------+----------------------+-----------------+ | lco-telescope | enx2cfda1c61ddf | 200.28.147.221 | 255.255.255.0 | 200.28.147.1 | 10.8.8.11 10.8.8.12 | lco.cl | +------------------+------------------+------------------+----------------+--------------------------+----------------------+-----------------+ | lco-visitor | | | 255.255.255.0 | | 10.8.8.11 10.8.8.12 | lco.cl | +------------------+------------------+------------------+----------------+--------------------------+----------------------+-----------------+ **For reference:** At last setup, the automatic DHCP-assigned configuration for ``www-ua`` was: - IP Address: ``128.196.208.35`` - Subnet Mask: ``255.255.252.0`` - Default Route: ``128.196.208.1`` - DNS: ``128.196.208.2 128.196.211.3 128.196.11.233 128.196.11.234`` exao2 ~~~~~ +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | connection name | device | IPv4 address | subnet mask | default route / gateway | DNS servers | search domains | +==================+==================+==================+================+==========================+============================================+=================+ | instrument | enx2cfda1c6db1a | 192.168.0.11 | 255.255.255.0 | 192.168.0.1 | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | www-ua | enx2cfda1c6db1b | 10.130.133.207 | 255.255.254.0 | 10.130.132.1 | 128.196.208.2 128.196.209.2 128.196.11.233 | as.arizona.edu | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | lco-telescope | enx2cfda1c6db1b | 200.28.147.222 | 255.255.255.0 | 200.28.147.1 | 10.8.8.11 10.8.8.12 | lco.cl | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | rtc-to-icc | enx00133b219c6e | 192.168.2.2 | 255.255.255.0 | n/a | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ ``instrument`` is a routerless network within the rack using a switch. ``rtc-to-icc`` is a direct NIC-to-NIC link between RTC and ICC. exao3 ~~~~~ +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | connection name | device | IPv4 address | subnet mask | default route / gateway | DNS servers | search domains | +==================+==================+==================+================+==========================+============================================+=================+ | instrument | enxd45d6407cb48 | 192.168.0.12 | 255.255.255.0 | 192.168.0.1 | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | www-ua | enxd45d6407cb49 | 10.130.133.208 | 255.255.254.0 | 10.130.132.1 | 128.196.208.2 128.196.209.2 128.196.11.233 | as.arizona.edu | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | lco-telescope | enxd45d6407cb49 | 200.28.147.223 | 255.255.255.0 | 200.28.147.1 | 10.8.8.11 10.8.8.12 | lco.cl | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | icc-to-rtc | enx6cb31152a245 | 192.168.2.3 | 255.255.255.0 | n/a | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | camsci1 | enx503eaa0ceeff | 192.168.101.2 | 255.255.255.0 | 192.168.101.1 | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ | camsci2 | enx503eaa0cf4cd | 192.168.102.2 | 255.255.255.0 | 192.168.102.1 | n/a | n/a | +------------------+------------------+------------------+----------------+--------------------------+--------------------------------------------+-----------------+ ``instrument`` is a routerless network within the rack using a switch. ``icc-to-rtc`` is a direct NIC-to-NIC link between RTC and ICC. The ``camsci1`` and ``camsci2`` networks are just direct connections from the Princeton Instruments cameras to their NICs. Hostnames --------- Each instrument computer has a ``/etc/hosts`` file installed with names and aliases for devices internal to MagAO-X. Changes to this file are made in `setup/steps/configure_etc_hosts.sh `__, and applied with ``provision.sh``. University of Arizona ~~~~~~~~~~~~~~~~~~~~~ While at the University of Arizona, the FQDN is ``.as.arizona.edu``. Only ``exao1`` has a publicly-routable IP address, while ``exao2`` and ``exao3`` live behind the NAT. Las Campanas Observatory ~~~~~~~~~~~~~~~~~~~~~~~~ While at LCO, the FQDN is ``.lco.cl``. All three instruments are accessible from the LCO-VISITORS wireless network and other usual places, but not from the outside internet. Time synchronization -------------------- Time synchronization depends on `chrony `__, configured at ``/etc/chrony/chrony.conf`` (Ubuntu 18.04) or ``/etc/chrony.conf`` (CentOS 7). Those files are updated by ``provision.sh`` according to the script in `setup/steps/configure_chrony.sh `__. The ICC and RTC take their time from AOC, which is configured to allow NTP queries from anyone in the ``192.168.0.0/24`` subnet. AOC, in turn gets its time from a combination of - ``lbtntp.as.arizona.edu`` - LBT / Steward Observatory NTP server (when in the lab) - ``ntp1.lco.cl`` - Las Campanas NTP server (when at the telescope) - ``ntp2.lco.cl`` - Backup Las Campanas NTP server (when at the telescope) - ``0.centos.pool.ntp.org`` — Alias for a pool of hosts that contribute to pool.ntp.org (whenever reachable) Troubleshooting ~~~~~~~~~~~~~~~ If you need to see how system time relates to network time on an instrument computer, run ``chronyc tracking``: :: $ chronyc tracking Reference ID : C0A8000A (exao1) Stratum : 3 Ref time (UTC) : Fri Nov 15 00:42:34 2019 System time : 0.000012438 seconds fast of NTP time Last offset : +0.000014364 seconds RMS offset : 0.000025598 seconds Frequency : 0.688 ppm fast Residual freq : +0.012 ppm Skew : 0.132 ppm Root delay : 0.000474306 seconds Root dispersion : 0.000256627 seconds Update interval : 130.4 seconds Leap status : Normal To force a (potentially discontinuous) time sync, ``sudo chronyc -a makestep``. To verify correct operation from RTC or ICC, use ``chronyc sources``: :: $ chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* exao1 2 6 377 25 +379ns[+1194ns] +/- 14ms If ``exao1`` is shown with a ``?`` in the second column or ``0`` in the ``Reach`` column, you may have firewalled traffic on the internal “instrument” interface. You can examine the configuration files in ``/etc/sysconfig/network-scripts/ifcfg-*`` and ensure that the interface corresponding to ``instrument`` in ``nmtui``/``nmcli`` has ``ZONE=trusted``. If it’s not any of that, consult the `chrony FAQ `__. To verify correct operation from the AOC end, ``sudo chronyc clients``: :: $ sudo chronyc clients [sudo] password for jlong: Hostname NTP Drop Int IntL Last Cmd Drop Int Last =============================================================================== localhost 0 0 - - - 49 0 11 16 exao2 92 0 6 - 21 0 0 - - exao3 27 0 6 - 16 0 0 - - If either exao2 or exao3 does not appear, ssh into them and verify ``chronyd`` has started… :: $ systemctl is-active chronyd active …ensure ``exao1`` is reachable via that name… :: $ ping exao1 PING exao1 (192.168.0.10) 56(84) bytes of data. 64 bytes from exao1 (192.168.0.10): icmp_seq=1 ttl=64 time=0.196 ms ... …and finally, consult the `chrony FAQ `__. Topology -------- Figure TODO