HPC Lab Machine Setup

Owned by Former user (Deleted)
Last updated: Sept 26, 2019 by Former user (Deleted)

The HPC Lab needs to use machines that are in pre-production and development stage, so a few quirks may be necessary to get them behaving like standard data centre machines.

Firmware / BIOS

D03/D05

BMC mode with HPM files (last resort)

Vendor firmware is signed and can be updated directly from the BMC web UI.

If you have the hpm file, navigate to System > Update Firmware, upload the hpm file and click "Upgrade".

EBL mode with FD files (deprecated)

If you have D0x.fd files and your current BIOS has an EBL option under Boot Manager, you can go into EBL and:

biosupdate 10.40.0.11 -u anonymous -p xxx -f D05.fd master

You need to download it and put into the local FTP server in Mr-Provisioner.

OS mode with Capsule files (preferred)

Download the Cap files and use fwupdate to update the firmware, but you have to have an OS installed.

root> apt|yum install -y fwupdate

root> fwupdate --info
# this will show an ID like {7978365d-7978-45fd-ad77-b27693cfe85b} and the current version

root> fwupdate --apply {...ID...} D0xFIRMWAREUPDATECAPSULE....Cap
# this will update the firmware

# reboot and try again
root> fwupdate --info
# this will show an ID and the current version, should be newer

BIOS mode with Capsule files (future)

Starting from 18.08, ERP firmware will allow capsule updates straight from the UEFI shell.


Use the latest firmware from releases.linaro.org.

Upstream firmware, based on Linaro's builds, to be found in estuary.org.

Centriq

Amberwings don't come ready to PXE boot automatically.

The network stack needs to be enabled in the BIOS screen, as described here.

ThunderX2

Upgrade the BIOS, as described in their internal documents (not public), using IPMI and the new hpm images.

A few comments on TX2 BIOS:

  • To be able to choose between booting on the internal PXE interface or the LAN one (e.g. Warewulf on the former, MrP on the latter), one needs to set one to be the 'default' boot order (which entails that it will PXE after a power cut off)
  • Flashing the BIOS with ipmitool sometimes does NOT seem to work (despite ipmitool saying its fine). In those cases you'll end up with boot entries missing, network interfaces not showing up, and other errors of the sort. All you need in those cases is to just reflash the BIOS, and it should do the trick. NOTE: During the reflash, you'll notice a mismatch between the (re)uploaded version and the one on the machine (whereas if the first flash had gone according to plan they'd be the same).
  • If it so happens that the ipmitool is spitting out errors of "version mismatch" and asking if you want to proceed, you need to update the BMC FW via the webinterface and using the .ima file (web paged is called "dual_firmware_upgrade"). Package 7.4 contains an .ima file that will reenable you to use the standard procedure via ipmitool.

Operating System

CentOS

AArch64 support in CentOS is done via AltArch. The repository and kernel images are available from their mirror, but that's always the latest release only.

NOTE: Make sure not to use Linaro's overlays with upstream CentOS, or the kernel will be overwritten and broken.

To install CentOS via PXE on Mr-Provisioner, use the following parameters:

Note: Use ttyS0 for D03 machines, and ttyAMA0 for all others.

To use the ERP releases, change in the following way:

Debian

To use the Debian ERP releases, just pick a release version from releases.linaro.org, for example, 17.12.

  • Kernel: arm64's linux
  • Initrd: arm64's initrd.gz
  • Preseed: Linaro's mini-install (as a starting point)
  • Kernel cmdline:earlycon console=ttyAMA0,115200

Note: Use ttyS0 for D03 machines, and ttyAMA0 for all others.

Grub Sub-Arch

Mr-Provisioner now supports two Arm64 Grub architectures: standard and forcing efiboot.

Amberwings require the efiboot flags, but the others work well with it, too. When in doubt, use efiboot and it should work.