Running a TFS Build Agent on your Raspberry PI / ARM based device

As I've mentioned a few times recently I've been pushing our use of ARM based devices at work for a while now. What started on Raspberry PI 2's &  3's as prototypes and early builds has developed into something much bigger. Which is great, however the way we've been writing and building code has got stuck in a bad process. For all of our .Net Core code we have a full CI pipeline which produces appropriate outputs for deploy to our devcies in an automated way. However in some of our more recent work we've needed to start compiling third party dependencies for our ARM based boards and even some C++ components.

It's always been on the "Todo:" list to ensure these dependencies and components got into our CI pipeline but with using various contractors and being "busy" things have gone too long with "works on my machine" builds and various network shares containing "OpenCV NEON optimised" and other such dependencies.

With the release of the Raspberry PI 4 last week and me ordering a 6 pack of these I decided it was time to not only compare how long things took to build on a PI 4 vs PI 3 / Compute Module but also to use 2 of my 6 as "ARM Build Agents" and add them to our TFS 2018 setup.

Below is a rough log of what I did, what I had to find out and hopefully how I got everything working :)

First Steps

The first steps were simply getting the latest version of Raspbian and installing this on my new Raspberry PI 4's and my bespoke Compute Module 3 setup. For both devices I wanted a very "lite" setup so chose the latest Raspbian Lite which is based on Debian Buster (at the time of writing this Buster isn't officially released but it's super close), it has no desktop which is perfect for our Build Agent. No use wasting resources on a full Desktop Experience, SSH is all you need ;). 

For the PI 4 I simply downloaded Raspbian and wrote the image to an SD Card but for my CM3 setup I wrote it directly to our eMMC module. For writing the image to the SD Card and our eMMC module I used ImageUSB as I find this to be a very good Windows Imaging solution, it looks dated but works really well and can do multiple devices at the same time. I use it to make images as much as I use it to deploy images.

With the images deployed I booted up both devices, changed the default password, enabled SSH via the raspi-config utility and then once all rebooted ensure all of the packages were fully up to date.

sudo apt update
sudo apt upgrade

Yes I am also doing all of this from Windows Terminal

Installing the TFS/Azure DevOps Build Agent

With the device up to date it was time to install the TFS / Azure DevOps Build Agent to the device. To deploy agents previously I've just navigated to the Agent Pools area within the TFS portal however dependant on your TFS version / if you are using Azure DevOps the URL can change. I highly recommend reading the docs for deploying a Linux Agent as it covers most things you need to know including authentication etc. However there was one thing I found a pain, it indicates the following:

• On the Get agent dialog box, click Linux.
• On the left pane, select the specific flavor. We offer x64 or ARM for most Linux distributions. We also offer a specific build for Red Hat Enterprise Linux 6.
• On the right pane, click the Download button.

But, this always gave me the x64 Linux agent, I needed a URL for the ARM build for my device. My first attempt to solve this was to swap the x64 part of the link to arm
From: https://vstsagentpackage.azureedge.net/agent/2.131.0/vsts-agent-linux-x64-2.131.0.tar.gz
To: From: https://vstsagentpackage.azureedge.net/agent/2.131.0/vsts-agent-linux-arm-2.131.0.tar.gz
But this didn't work. :(

Next I decided to go find the source of the agent on GitHub this quite happily shows ARM Linux builds as well as install information.

However all of the links go back to the same documentation with no actual links! At this point I almost decided to give up but the Azure Pipeline badges made me thing it has to be outputting them somewhere.... so I went digging :)

Hurrah Releases!

Look at that lovely file name with the build number

This then meant I could do some substution and get an agent url of https://vstsagentpackage.azureedge.net/agent/2.154.0/vsts-agent-linux-arm-2.154.0.tar.gz Hurrah a working link. Now it's time to get it onto the devices.  You could download the package and SCP it across to the device but that feels like hassle. Instead I just used wget to download the package straight onto the device:

wget https://vstsagentpackage.azureedge.net/agent/2.154.0/vsts-agent-linux-arm-2.154.0.tar.gz

With this done I could follow the previous instructions but updated the version numbers:

mkdir agent && cd agent
tar zxvf ~/vsts-agent-linux-arm-2.154.0.tar.gz

Now with this extracted I came to configure the agent with ./config.sh but found I was missing dependencies so followed the instructions and ran sudo ./bin/installdependencies.sh

This however currently errors on Buster due to missing dependencies , I came to find all the dependency updates and potentially submit a PR back to update the script but fortunately found ItalyPaleAle had already done this so I just used wget to get the script directly and then run it. (Although I've provided the commands below it's worth double checking the script before executing it as sudo to ensure it's not got any nasties in it :) )

wget https://raw.githubusercontent.com/microsoft/azure-pipelines-agent/bb3afca3a2b1032842073c80bb5f1cfd0516665f/src/Misc/layoutbin/installdependencies.sh
chmod +x installdependencies.sh && sudo ./installdependencies.sh

With the dependencies installed i then reran ./config.sh but again hit another error:

./bin/System.Net.Http.Native.so: /usr/lib/arm-linux-gnueabihf/libcurl.so.4: version `CURL_OPENSSL_3' not found (required by ./bin/System.Net.Http.Native.so)
Dependencies is missing for Dotnet Core 2.1

Fail

So, more Googling led me to find this GitHub issue for .Net Core in which our friend ItalyPaleAle has also added a comment!

So what next....

Essentially, I went back to the previous version of Raspbian and gave up on supporting the Raspberry PI 4 at this time. Back on Rasbian Stretch we can successfully run config.sh.

As part of the config you have to accept a End User licence agreement and then provide the TFS/Azure DevOps url. Use the documentation on how to find the URL to use.  You then have to provide a Personal Access Token for the Build Agent to use to authenticate with your TFS/Azure Dev Ops environment, follow the documentation on how to set this up and with what permissions.

Here I hit another issue, I kept getting Resource not available errors whilst trying to authenticate

Bit of googling led me to this GitHub issue, Annoyingly it indicates a workaround not a fix. I had to temporarily disable HTTP basic auth within IIS, get the agent registered and then re-enable it as we use basic auth as a workaround for our NuGet issue ;(

With this all finally configured our agent appears within TFS

I then configured to run the agent as a systemd service so it auto starts etc

With this complete the agent is ready waiting for jobs.

Additional Dependencies / Configuration

As things stand the Agent is running but it's not particularly useful. It doesn't have git or any of our build dependencies. The next step was to install these dependencies.

sudo apt install git

etc

and then finally....

Oh yeah! Running builds on the pi's

Success :)

I will update this post once the Buster is fully supported and ensure there's no other steps required. You can see it wasn't a straight forward setup as I hoped and this process definitely made me appreciate the potential value of using Microsoft Hosted Agents but overall everything is running and hopefully giving the team some value and reliable builds :)

Hope this helps.