Coolant System Backflush

After a period of time (months), the glycol-based cooling system accumulates the byproduct of some kind of organism, maybe a type of slime mold, and the flow of glycol becomes restricted. This procedure should be done every 4-6 months to prevent high GPU and/or high CPU temps.

The filter basket needs to be cleaned before starting this procedure.

Procedure

  1. Required Tools

    • 2x CPU backflush jigs

    • 2x GPU backflush jigs (smaller hose diam.)

    • paper towels

    • large flathead screwdriver (rack panel removal)

  2. Initial Conditions.

    • RTC, ICC, and liquid cooled cameras (camwfs, camsci1, camsci2) off.

    • instCool on.

    • filter basket has been cleaned and reinstalled.

  3. Disconnect the coolant hoses at the MagAO-X inlet and connect them together to form a closed loop. This is to maximize flow rate though the electronics rack.

4. Remove the large cover plate on the input side of the electronics rack (opposite to MagAO-X) to expose the cooler control panel.

../../_images/cpu_gpu_cooling.jpeg

View after removing the large input side cover plate

  1. Note coolant flow in instCool.status.flow_rate and on the cooler control panel. Make note of the initial average flow rate to gauge progress during the backflush.

  2. Power off instCool with the rocker switch. Do this locally, as opposed to with pwrGUI, for later leak checks.

  3. Locate the hoses connected to each device to be backflushed. The GPUs are plumbed in series, so they must be backflushed one-by-one to avoid pushing large masses into the restrictive heat exchangers on each GPU.

    ../../_images/inside_rack_hoses.jpeg

    Example ICC CPU hose loop looking inside the rack from the front.

Warning

The quick-disconnects are well-engineered and normally leak-proof, but protect the instrument computers with paper towels under the hoses for good measure.

  1. Disconnect the two coolant hoses to one device and use the backflush jigs to reverse the flow of coolant to that specific device.

../../_images/backflush_jigs_og.jpeg

The jigs are composed of a male<->male and female<->female configuration for flow reversal.

../../_images/backflush_jigs_new.jpeg

The GPU hoses have a smaller diameter, so they require the jigs that feature a step down in connector size.

  1. Power on instCool with the rocker switch and let the coolant flow for ~2 minutes. Make notes on the flow rate with time. Power OFF :dev:`instCool`to lower cooling system pressure. Remove the jigs and reconnect the rack coolant hoses to restore the nominal configuration.

../../_images/instCool_display.jpg

Flow rate should improve after a minute or so of backflushing.

  1. Repeat steps 8 and 9 for each device that needs backflushing.

  2. After backflushing, double check each hose connection for leaks when instCool is ON.

  3. The filter basket needs to be cleaned after the backflushing procedure as per the guide.

The filter -will- be clogged after backflushing, ensure that it’s clean before returning to normal ops.

  1. Finish up by reinstalling the input-side cover plate on the electronics rack and reconnecting the coolant hoses at the MagAO-X inlet.

Leak Check

The job is done when instCool is powered back on and all external connections are showing no leaks.