diff options
| -rw-r--r-- | .gitlab-ci/bare-metal/README.md | 88 | ||||
| -rwxr-xr-x | .gitlab-ci/bare-metal/cros-servo.sh | 97 | ||||
| -rwxr-xr-x | .gitlab-ci/bare-metal/fastboot.sh | 54 | ||||
| -rw-r--r-- | .gitlab-ci/bare-metal/rootfs-setup.sh | 55 | ||||
| -rwxr-xr-x | .gitlab-ci/bare-metal/write-serial.py | 11 |
5 files changed, 232 insertions, 73 deletions
diff --git a/.gitlab-ci/bare-metal/README.md b/.gitlab-ci/bare-metal/README.md index 3bbfab11cfb..843d168c455 100644 --- a/.gitlab-ci/bare-metal/README.md +++ b/.gitlab-ci/bare-metal/README.md @@ -3,40 +3,76 @@ Testing Mesa with gitlab-runner running on the devices being tested (DUTs) proved to be too unstable, so this set of scripts is for running Mesa testing on bare-metal boards connected to a separate -system using gitlab-runner. Currently only "fastboot" devices are -supported. +system using gitlab-runner. Currently only "fastboot" and "ChromeOS +Servo" devices are supported. In comparison with LAVA, this doesn't involve maintaining a separate webservice with its own job scheduler and replicating jobs between the two. It also places more of the board support in git, instead of -webservice configuration. Most importantly, it doesn't download the -rootfs as artifacts on each job, so we can avoid traffic to -freedesktop.org. On the other hand, the serial interactions and -bootloader support are more primitive. +webservice configuration. On the other hand, the serial interactions +and bootloader support are more primitive. -## Requirements +## Requirements (fastboot) This testing requires power control of the DUTs by the gitlab-runner machine, since this is what we use to reset the system and get back to a pristine state at the start of testing. -We require access to the console output from the gitlb-runner system, -since that is how we get the final results back from te tests. You +We require access to the console output from the gitlab-runner system, +since that is how we get the final results back from the tests. You should probably have the console on a serial connection, so that you can see bootloader progress. The boards need to be able to have a kernel/initramfs supplied by the gitlab-runner system, since the initramfs is what contains the Mesa -testing payload. Currently only "fastboot" devices are supported. +testing payload. The boards should have networking, so that (in a future iteration of this code) we can extract the dEQP .xml results to artifacts on gitlab. +## Requirements (servo) + +For servo-connected boards, we can use the EC connection for power +control to reboot the board. However, loading a kernel is not as easy +as fastboot, so we assume your bootloader can do TFTP, and that your +gitlab-runner mounts the runner's tftp directory specific to the board +at /tftp in the container. + +Since we're going the TFTP route, we also use NFS root. This avoids +packing the rootfs and sending it to the board as a ramdisk, which +means we can support larger rootfses (for piglit or tracie testing), +at the cost of needing more storage on the runner. + +Telling the board about where its TFTP and NFS should come from is +done using dnsmasq on the runner host. For example, this snippet in +the dnsmasq.conf.d in the google farm, with the gitlab-runner host we +call "servo". + +``` +dhcp-host=1c:69:7a:0d:a3:d3,10.42.0.10,set:servo + +# Fixed dhcp addresses for my sanity, and setting a tag for +# specializing other DHCP options +dhcp-host=a0:ce:c8:c8:d9:5d,10.42.0.11,set:cheza1 +dhcp-host=a0:ce:c8:c8:d8:81,10.42.0.12,set:cheza2 + +# Specify the next server, watch out for the double ',,'. The +# filename didn't seem to get picked up by the bootloader, so we use +# tftp-unique-root and mount directories like +# /srv/tftp/10.42.0.11/jwerner/cheza as /tftp in the job containers. +tftp-unique-root +dhcp-boot=tag:cheza1,cheza1/vmlinuz,,10.42.0.10 +dhcp-boot=tag:cheza2,cheza2/vmlinuz,,10.42.0.10 + +dhcp-option=tag:cheza1,option:root-path,/srv/nfs/cheza1 +dhcp-option=tag:cheza2,option:root-path,/srv/nfs/cheza2 +``` + ## Setup Each board will be registered in fd.o gitlab. You'll want something -like this to register: +like this to register a fastboot board: ``` sudo gitlab-runner register \ @@ -52,27 +88,35 @@ sudo gitlab-runner register \ --non-interactive ``` +For a servo board, you'll need to also volume mount the board's NFS +root dir at /nfs and TFTP kernel directory at /tftp. + The registration token has to come from a fd.o gitlab admin going to https://gitlab.freedesktop.org/admin/runners -The name scheme for Google's lab is google-freedreno-boardname-nn, and -our tag is google-freedreno-db410c. The tag is what identifies a -board type so that board-specific jobs can be dispatched into that -pool. +The name scheme for Google's lab is google-freedreno-boardname-n, and +our tag is something like google-freedreno-db410c. The tag is what +identifies a board type so that board-specific jobs can be dispatched +into that pool. We need privileged mode and the /dev bind mount in order to get at the serial console and fastboot USB devices (--device arguments don't apply to devices that show up after container start, which is the case -with fastboot). We use host network mode so that we can (in the -future) spin up a server to collect XML results. - -Once you've added your boards, you're going to need to specify the -board-specific env vars, adding something like this `environment` line -to each runner in `/etc/gitlab-runner/config.toml` +with fastboot, and the servo serial devices are acctually links to +/dev/pts). We use host network mode so that we can (in the future) +spin up a server to collect XML results for fastboot. + +Once you've added your boards, you're going to need to add a little +more customization in `/etc/gitlab-runner/config.toml`. First, add +`concurrent = <number of boards>` at the top ("we should have up to +this many jobs running managed by this gitlab-runner"). Then for each +board's runner, set `limit = 1` ("only 1 job served by this board at a +time"). Finally, add the board-specific environment variables +required by your bare-metal script, something like: ``` [[runners]] - name = "google-freedreno-db410c-01" + name = "google-freedreno-db410c-1" environment = ["BM_SERIAL=/dev/ttyDB410c8", "BM_POWERUP=google-power-up.sh 8", "BM_FASTBOOT_SERIAL=15e9e390"] ``` diff --git a/.gitlab-ci/bare-metal/cros-servo.sh b/.gitlab-ci/bare-metal/cros-servo.sh new file mode 100755 index 00000000000..b67c1f2ad65 --- /dev/null +++ b/.gitlab-ci/bare-metal/cros-servo.sh @@ -0,0 +1,97 @@ +#!/bin/bash + +# Boot script for Chrome OS devices attached to a servo debug connector, using +# NFS and TFTP to boot. + +# We're run from the root of the repo, make a helper var for our paths +BM=$CI_PROJECT_DIR/.gitlab-ci/bare-metal + +# Runner config checks +if [ -z "$BM_SERIAL" ]; then + echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment" + echo "This is the CPU serial device." + exit 1 +fi + +if [ -z "$BM_SERIAL_EC" ]; then + echo "Must set BM_SERIAL in your gitlab-runner config.toml [[runners]] environment" + echo "This is the EC serial device for controlling board power" + exit 1 +fi + +if [ ! -d /nfs ]; then + echo "NFS rootfs directory needs to be mounted at /nfs by the gitlab runner" + exit 1 +fi + +if [ ! -d /tftp ]; then + echo "TFTP directory for this board needs to be mounted at /tftp by the gitlab runner" + exit 1 +fi + +# job config checks +if [ -z "$BM_KERNEL" ]; then + echo "Must set BM_KERNEL to your board's kernel FIT image" + exit 1 +fi + +if [ -z "$BM_ROOTFS" ]; then + echo "Must set BM_ROOTFS to your board's rootfs directory in the job's variables" + exit 1 +fi + +if [ -z "$BM_CMDLINE" ]; then + echo "Must set BM_CMDLINE to your board's kernel command line arguments" + exit 1 +fi + +set -ex + +# Create the rootfs in the NFS directory. rm to make sure it's in a pristine +# state, since it's volume-mounted on the host. +rm -rf /nfs/* +mkdir -p /nfs/results +. $BM/rootfs-setup.sh /nfs + +# Set up the TFTP kernel/cmdline. When we support more than one board with +# this method, we'll need to do some check on the runner name or something. +rm -rf /tftp/* +cp $BM_KERNEL /tftp/vmlinuz +echo "$BM_CMDLINE" > /tftp/cmdline + +# Start watching serials, and power up the device. +$BM/serial-buffer.py $BM_SERIAL_EC | tee serial-ec-output.txt | sed -u 's|^|SERIAL-EC> |g' & +$BM/serial-buffer.py $BM_SERIAL | tee serial-output.txt | sed -u 's|^|SERIAL-CPU> |g' & +while [ ! -e serial-output.txt ]; do + sleep 1 +done +# Flush any partial commands in the EC's prompt, then ask for a reboot. +$BM/write-serial.py $BM_SERIAL_EC "" +$BM/write-serial.py $BM_SERIAL_EC reboot + +# This is emitted right when the bootloader pauses to check for input. Emit a +# ^N character to request network boot, because we don't have a +# direct-to-netboot firmware on cheza. +$BM/expect-output.sh serial-output.txt "load_archive: loading locale_en.bin" +$BM/write-serial.py $BM_SERIAL `printf '\016'` + +# Wait for the device to complete the deqp run +$BM/expect-output.sh serial-output.txt "DEQP RESULT" + +# power down the CPU on the device +$BM/write-serial.py $BM_SERIAL_EC 'power off' + +set -ex + +# Bring artifacts back from the NFS dir to the build dir where gitlab-runner +# will look for them. Note that results/ may already exist, so be careful +# with cp. +mkdir -p results +cp -Rp /nfs/results/. results/ + +set +e +if grep -q "DEQP RESULT: pass" serial-output.txt; then + exit 0 +else + exit 1 +fi diff --git a/.gitlab-ci/bare-metal/fastboot.sh b/.gitlab-ci/bare-metal/fastboot.sh index 221d8efc26e..124309e8abe 100755 --- a/.gitlab-ci/bare-metal/fastboot.sh +++ b/.gitlab-ci/bare-metal/fastboot.sh @@ -44,57 +44,9 @@ fi set -ex -# Copy the rootfs to a temporary for our setup, as I believe changes to the -# container can end up impacting future runs. -cp -Rp $BM_ROOTFS rootfs - -# Set up the init script that brings up the system. -cp $BM/init.sh rootfs/init - -set +x -# Pass through relevant env vars from the gitlab job to the baremetal init script -touch rootfs/set-job-env-vars.sh -chmod +x rootfs/set-job-env-vars.sh -for var in \ - CI_COMMIT_BRANCH \ - CI_COMMIT_TITLE \ - CI_JOB_ID \ - CI_JOB_URL \ - CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \ - CI_MERGE_REQUEST_TITLE \ - CI_NODE_INDEX \ - CI_NODE_TOTAL \ - CI_PIPELINE_ID \ - CI_RUNNER_DESCRIPTION \ - DEQP_CASELIST_FILTER \ - DEQP_EXPECTED_RENDERER \ - DEQP_NO_SAVE_RESULTS \ - DEQP_PARALLEL \ - DEQP_RUN_SUFFIX \ - DEQP_VER \ - FD_MESA_DEBUG \ - FLAKES_CHANNEL \ - IR3_SHADER_DEBUG \ - NIR_VALIDATE \ - ; do - val=`echo ${!var} | sed 's|"||g'` - echo "export $var=\"${val}\"" >> rootfs/set-job-env-vars.sh -done -echo "Variables passed through:" -cat rootfs/set-job-env-vars.sh -set -x - -# Add the Mesa drivers we built, and make a consistent symlink to them. -mkdir -p rootfs/$CI_PROJECT_DIR -tar -C rootfs/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar -ln -sf $CI_PROJECT_DIR/install rootfs/install - -# Copy the deqp runner script and metadata. -cp .gitlab-ci/deqp-runner.sh rootfs/deqp/. -cp .gitlab-ci/$DEQP_SKIPS rootfs/$CI_PROJECT_DIR/install/deqp-skips.txt -if [ -n "$DEQP_EXPECTED_FAILS" ]; then - cp .gitlab-ci/$DEQP_EXPECTED_FAILS rootfs/$CI_PROJECT_DIR/install/deqp-expected-fails.txt -fi +# Create the rootfs in a temp dir +mkdir rootfs +. .gitlab-ci/bare-metal/rootfs-setup.sh rootfs # Finally, pack it up into a cpio rootfs. pushd rootfs diff --git a/.gitlab-ci/bare-metal/rootfs-setup.sh b/.gitlab-ci/bare-metal/rootfs-setup.sh new file mode 100644 index 00000000000..f8c9fa42ba0 --- /dev/null +++ b/.gitlab-ci/bare-metal/rootfs-setup.sh @@ -0,0 +1,55 @@ +#!/bin/bash + +rootfs_dst=$1 + +# Copy the rootfs to a temporary for our setup, as I believe changes to the +# container can end up impacting future runs. +cp -Rp $BM_ROOTFS/. $rootfs_dst + +# Set up the init script that brings up the system. +cp $BM/init.sh $rootfs_dst/init + +set +x +# Pass through relevant env vars from the gitlab job to the baremetal init script +touch $rootfs_dst/set-job-env-vars.sh +chmod +x $rootfs_dst/set-job-env-vars.sh +for var in \ + CI_COMMIT_BRANCH \ + CI_COMMIT_TITLE \ + CI_JOB_ID \ + CI_JOB_URL \ + CI_MERGE_REQUEST_SOURCE_BRANCH_NAME \ + CI_MERGE_REQUEST_TITLE \ + CI_NODE_INDEX \ + CI_NODE_TOTAL \ + CI_PIPELINE_ID \ + CI_RUNNER_DESCRIPTION \ + DEQP_CASELIST_FILTER \ + DEQP_EXPECTED_RENDERER \ + DEQP_NO_SAVE_RESULTS \ + DEQP_PARALLEL \ + DEQP_RUN_SUFFIX \ + DEQP_VER \ + FD_MESA_DEBUG \ + FLAKES_CHANNEL \ + IR3_SHADER_DEBUG \ + NIR_VALIDATE \ + ; do + val=`echo ${!var} | sed 's|"||g'` + echo "export $var=\"${val}\"" >> $rootfs_dst/set-job-env-vars.sh +done +echo "Variables passed through:" +cat $rootfs_dst/set-job-env-vars.sh +set -x + +# Add the Mesa drivers we built, and make a consistent symlink to them. +mkdir -p $rootfs_dst/$CI_PROJECT_DIR +tar -C $rootfs_dst/$CI_PROJECT_DIR/ -xf $CI_PROJECT_DIR/artifacts/install.tar +ln -sf $CI_PROJECT_DIR/install $rootfs_dst/install + +# Copy the deqp runner script and metadata. +cp .gitlab-ci/deqp-runner.sh $rootfs_dst/deqp/ +cp .gitlab-ci/$DEQP_SKIPS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-skips.txt +if [ -n "$DEQP_EXPECTED_FAILS" ]; then + cp .gitlab-ci/$DEQP_EXPECTED_FAILS $rootfs_dst/$CI_PROJECT_DIR/install/deqp-expected-fails.txt +fi diff --git a/.gitlab-ci/bare-metal/write-serial.py b/.gitlab-ci/bare-metal/write-serial.py new file mode 100755 index 00000000000..6a57ea754af --- /dev/null +++ b/.gitlab-ci/bare-metal/write-serial.py @@ -0,0 +1,11 @@ +#!/usr/bin/python3 + +import sys +import serial + +dev = sys.argv[1] +command = sys.argv[2] + '\n' + +ser = serial.Serial(dev, 115200, timeout=5) +ser.write(command.encode()) +ser.close() |