Build your first image

Migration notice

After more than 2 years of development, we have grown beyond what rbuild's underlying dependencies are capable of. We have switched to a lower-level tool and completely restructured the project.

Please use rsdk for the new project, which is meant to be compatible with existing rbuild systems. rbuild is now in maintenance mode.

Currently, due to our dependency on debos, we can only run rbuild on x86 based system. If you are building in a virtual machine, you need to enable nested virtualization, as KVM is required for building the image.

Debian 12 / Ubuntu 22.04

sudo apt update
sudo apt install -y git

# Podman (recommended)
sudo apt install -y podman podman-docker
sudo touch /etc/containers/nodocker
# Docker
#sudo apt install -y docker.io
#sudo adduser $USER docker
#sudo reboot

Check out the code

git clone https://github.com/radxa-repo/rbuild.git

Build your first image

Once the repo is cloned on your machine, you can run rbuild without any arguments to check the help message:

cd rbuild
./rbuild

Most options listed in the help messages are targetting at developers. If you only want to build an image locally, you can run rbuild with only the required arguments:

# Build radxa-cm3-sodimm-io image with default OS (currently Debian Bullseye) and flavor (CLI)
./rbuild radxa-cm3-sodimm-io
# Build rock-5b Debian image with KDE
./rbuild rock-5b kde

Supported products, suites, and flavors are listed at the end of the help message.

Note

Radxa supports and recommends Debian on all Radxa products.

Building Ubuntu requires special instruction.

Running from GitHub Workflows

Please check out our GitHub workflows as an example.

Set some arguments as default on

You can create .rbuild-config file under the project folder, which will be sourced by the shell before rbuild processes the command line arguments. This is a convenient place to override the default behavior of rbuild to better suit to a development workflow.

Below is an example used in our work environment:

CONTAINER_BACKEND="podman"                          # podman allows multiple users
                                                    # to run rbuild on the same build server
REPO_PREFIX="-test"                                 # Build against the latest code
RBUILD_DISTRO_MIRROR="http://apt.vamrs.com"         # Use internal apt mirror
RBUILD_RADXA_MIRROR="http://apt.vamrs.com/rbuild-"  # Use internal apt mirror

Use apt mirrors to speed up build

If you have set up an apt local mirror (for example, using AptCacheNG), you can edit the config file to use those mirrors to build your image.

Please be aware that the mirror will be used as image's apt repo as well, so it should be used only for local development build.

Example below uses an internal mirror, that is not accessible to the public.

$ cat ~/rbuild/.rbuild-config 
RBUILD_DISTRO_MIRROR="http://apt.vamrs.com"
RBUILD_RADXA_MIRROR="http://apt.vamrs.com/rbuild-"

rbuild will automatically complete the mirror URL based on the build configurations.

You can also define them with the command line arguments -m|--mirror and -M.

Use locally built kernel and firmware

It is very common during the bring-up stage that the kernel and the firmware are evolving rapidly. This makes fetching those packages from apt repo very cumbersome. rbuild tailors to this use case, and allow the use of locally built kernel and firmware Deb package.

One thing we specifically do not allow is to install arbitrary packages with command arguments. The reason is that this makes tracking the image's origin more difficult. Consider the options listed here as exceptions.

Proper way for adding product specific package should be either update yaml templates, or via radxa-profiles package.

`-c|--custom` flag

Consider this: you build linux-latest and u-boot-latest packages with bsp and want to use them for your rbuild image. As long as bsp and rbuild are located under the same parent folder, you can use -c flag to consume both packages at once:

./rbuild -c latest radxa-zero

When we say "under the same parent folder", we mean something like /home/user/bsp and /home/user/rbuild, where both are under the same /home/user folder. -c flag will search bsp repo with the same directory level. However, if you copied them to rbuild folder, it will also be searched.

-c flag's argument is the same bsp profile name, so in the above example, we are able to use one flag to match both kernel and firmware packages. However, -c flag will not complain if there is no matching package, so you don't have to build both the kernel and firmware in order to use this flag.

You can even specify -c flag multiple times, if your kernel and firmware are using different profiles. For example, many Rockchip products uses linux-rockchip and u-boot-latest. You can simply run the following command for those products:

./rbuild -c rockchip -c latest rock-4c-plus

However, there is an issue. What if we have both linux-rockchip and linux-latest packages built in bsp? Currently, the second -c flag will override the previous choice, so you end up with both linux-latest and u-boot-latest for image above.

If this is not intended, you can either delete unused packages from your bsp repo, or using the below flags to manually specify a package.

`-k|--kernel` and `-f|--firmware` flags

Under the hood, -c flag calls the functionalilty of -k and -f flages. They take a path to a build Deb packages, so there is no second guessing.

Unlike -c, those flages will throw out error if the specified package does not exist, which is to be expected from a lower level feature.

About Ubuntu

Current status

While Ubuntu is a build target for rbuild, it is not officially supported by Radxa for use on our products. The reason is multifold:

Starting from Ubuntu 21.10, all of its packages are compressed with Zstd. We are currently using the official Debos docker image, which is based on Debian, and does not support Zstd compressed package. Attempts to create an Ubuntu based docker image are getting nowhere, so currently one has to both use Ubuntu as the build host and disable the container build for rbuild to work, which greatly limits the developer's choice of OS and build reproducibility.
Rockchip only provides Debian SDK, and for some dependencies the package names and/or versions are different between Debian and Ubuntu, meaning they cannot be installed as-is. This cannot be simply fixed by changing the control file pointing to a different package, as packaged binaries are hardcoded to some specific version of dynamic libraries, and there might be incompatible ABI changes between the two OS. Recompilation and repackaging are required to properly fix this issue, but they can be time-consuming, and sometimes the necessary code for those is not available.

For those reasons, Radxa has historically only provided the Ubuntu CLI image with no vendor hardware enablement packages. With the release of rbuild, we now dropped Ubuntu CLI as an officially supported system entirely, and recommend our users to use Debian CLI instead. The only exception is that some users want to run ROS on our products, which requires Ubuntu.

Build Ubuntu

To build Ubuntu, there are additional requirements beyond what was listed in Build your first image:

The host should run Ubuntu.
The host Ubuntu version should be greater or equal to the version you plan to build.
You should install debos on your system: sudo apt-get update && sudo apt-get install -y debos.
Replace any ./rbuild command with sudo ./rbuild --native-build like what we did in our GitHub Action (which runs on Ubuntu runner).

General architecture

rbuild is the final stage of our image-building pipeline. However, before it can be triggered, many subcomponents and tasks have to be completed and released first.

Submit code changes

The following repositories contain the source code, so the related commits should be pushed in the repo first before the planned release:

Kernel
U-Boot
Overlays
and any repositories that contain Makefile under radxa-pkg

Release Debian packages

For packages under radxa-pkg, once changes are made, please run make dch command to create a new changelog entry.

Edit debian/changlog accordingly, then change UNRELEASED to stable. You should then create a commit containing only this change, with the commit title Release x.y.z.

It is recommended to run make deb after you commit your changelog edit, so the package can be tested by lintian for common pitfalls. We treat warning as error, so please fix them, instead of suppressing them.

GitHub Workflows will then detect this new version, and create a new GitHub Release with the build artifacts. You can manually trigger the workflow from the website or within project folder using following command:

make release

Kernel and U-Boot's package repo under radxa-pkg needs to be reworked to follow the above release method. Currently the workflow will create a new release if VERSION file is updated. You can also manually trigger the workflow as above.

Before releasing the Kernel package, overlay.sh needs to be updated to pointing at the latest overlays commit. This is to pin overlays version with bsp version.

Update apt repos

While testing repos will sync daily with the latest package releases, the production repos require manual updating, so unverified software will slip past testing. At least that's the plan. Currently, production repos also pull the latest packages.

There are 2 workflows to update the apt repo. update.yml will fetch any new packages, and update the index files. There is no downtime during the update, so this should be preferred for updating small packages.

The other workflow reset.yml will first clear the branch history, before pulling packages. This is because the normal update.yml won't delete old packages, and the naive approach is not suitable since some systems require an older version of the package (which should be added explicitly). This should be the one to use if there is a new kernel or U-Boot package.

Below is an bash example to trigger apt repo update:

set -euo pipefail
for i in buster bullseye focal jammy
do
    gh workflow run --repo radxa-repo/$i update.yml
done

Depending on which workflow you use, you will see 1 or 2 completed pages build and deployment runs in Actions history, which indicates the apt repo has been updated.

Trigger image build

Once apt repo is updated. We can trigger RC image build. This is also done using workflows:

set -euo pipefail
for i in rock-3c radxa-cm3-sodimm-io
do
    gh workflow run --repo radxa-build/$i build.yml
done

By default, the release will be marked as pre-release. Once it passes the internal testing, it can be promoted as the latest official release.

Reproduce released image

While we recommend everyone keep their system up-to-date, sometimes, the customer develops their solution on one of our previously released images. Since they have validated the use case on that specific release, they want to reproduce images based on that specific release.

In this article we will show you how to reproduce rock-3c_debian_bullseye_xfce_b27.img.xz as an example. Please also have git and gh installed on your system.

Get build time information from released image

Every released rbuild images contains 2 files describing their build time environment. They are /etc/radxa_image_fingerprint for the build system, and /etc/radxa_apt_snapshot for the then-available packages on Radxa official APT repo.

Check the content from a running system

radxa@rock-3c:~$ cat /etc/radxa_image_fingerprint
RBUILD_BUILD_DATE='Tue, 21 Mar 2023 07:06:37 +0000'
RBUILD_REVISION='6d211a5998fdfc9f58fe7b9a2f507ec8c28199a2'
RBUILD_COMMAND='./rbuild --timestamp=b27 --compress --native-build --shrink --root-override rock-3c bullseye xfce'
RBUILD_KERNEL='linux-image-4.19.193-1-rk356x'
RBUILD_KERNEL_VERSION='4.19.193-1-41024583a'
RBUILD_UBOOT='u-boot-rk356x'
RBUILD_UBOOT_VERSION='2017.09-1-15c53b0'
radxa@rock-3c:~$ cat /etc/radxa_apt_snapshot 
{
  "libreelec-alsa-utils": "10.0.2-1",
  "radxa-otgutils": "0.2.1",
  "rsetup": "0.3.13",
  ...
}

Check the content from a disk image

You can use following commands to check the system info without a running device:

sudo apt update
sudo apt install multipath-tools
sudo kpartx -a system.img
# Check with `lsblk` to find the last loop device partition
# In this example we assume it is loop0p3
sudo mount /dev/mapper/loop0p3 /mnt
cat /mnt/etc/radxa_image_fingerprint
sudo umount /mnt
sudo kpartx -d system.img

Create a custom apt repo

As shown in RBUILD_COMMAND above, our image is based on Debian Bullseye. We will have to create a custom apt repo for rbuild based on radxa_apt_snapshot, since the official Radxa apt repo is likely to have newer packages.

In this article we will only fork the existing Radxa apt repo. If you want to create a apt repo from scrach, please check Create apt repo from scratch.

First, make sure we have logged in with gh:

[excalibur@yuntian reproduce]$ gh auth status
github.com
  ✓ Logged in to github.com as RadxaYuntian (/home/excalibur/.config/gh/hosts.yml)
  ...

If the account is incorrect, we can use gh auth logout; gh auth login to authenticate with the desired account.

We can now create a fork of the apt repo. Recall we need one targetting bullseye:

RELEASE=bullseye
GITHUB_NAMESPACE=your_account_or_orginazation
gh repo fork radxa-repo/$RELEASE --default-branch-only <<< "n"
gh repo clone $GITHUB_NAMESPACE/$RELEASE
cd $RELEASE
nano pkgs.lock # paste the content of /etc/radxa_apt_snapshot
git add pkgs.lock
git commit -m "Add pkgs.lock"

We now need to create a custom apt signing key. Proper key management is beyond the scope of this article, so we will create

GPG_KEY_EMAIL=testing@repo.com
sed -i "s/dev@radxa.com/$GPG_KEY_EMAIL/" .freight.conf
git add .freight.conf
git commit -m "Update signing key"
# Below key is for testing only, please follow Debian keyring policy for production usage
gpg --batch --quick-gen-key --passphrase "" $GPG_KEY_EMAIL
gpg --armor --export-secret-keys $GPG_KEY_EMAIL | gh secret set --repo $GITHUB_NAMESPACE/$RELEASE GPG_KEY

We can finally start populating our apt repo:

git switch -c gh-pages
git push --all
gh repo set-default $GITHUB_NAMESPACE/$RELEASE
gh workflow disable static.yml
gh workflow run update.yml

We can check the workflow status with gh run list. After a while, we should see 2 successful runs for pages-build-deployment workflow:

[excalibur@yuntian bullseye]$ gh workflow view pages-build-deployment
pages-build-deployment - pages-build-deployment
ID: 52533629

Total runs 2
Recent runs
✓  pages build and deployment  pages-build-deployment  gh-pages  dynamic  4538811569
✓  pages build and deployment  pages-build-deployment  gh-pages  dynamic  4538785650

To see more runs for this workflow, try: gh run list --workflow pages-build-deployment
To see the YAML for this workflow, try: gh workflow view pages-build-deployment --yaml

The first run is created when we pushed gh-pages branch (which also enabled workflow on forked repo). The second run is triggered after packages has been fetched, and it is this one that makes our apt repo functioning.

Prepare `rbuild` for reproducing released image

Recall rbuild commit ID was saved in /etc/radxa_image_fingerprint, we will checkout at this exact commit:

RBUILD_REVISION='6d211a5998fdfc9f58fe7b9a2f507ec8c28199a2'
cd .. # leave bullseye repo
gh repo clone radxa-repo/rbuild
cd rbuild
git switch --detach $RBUILD_REVISION

We also need to edit common/scripts/add_radxa_repo.yaml to switch the default repo URL and fetch keyring from apt repo:

[excalibur@yuntian rbuild]$ git diff common/scripts/add_radxa_repo.yaml
diff --git a/common/scripts/add_radxa_repo.yaml b/common/scripts/add_radxa_repo.yaml
index 448857c..f25a10a 100644
--- a/common/scripts/add_radxa_repo.yaml
+++ b/common/scripts/add_radxa_repo.yaml
@@ -1,4 +1,4 @@
-{{- $radxa_mirror := "https://radxa-repo.github.io/" -}}
+{{- $radxa_mirror := "https://radxayuntian.github.io/" -}}
 
 {{- $origin := .origin -}}
 {{- $suite := .suite -}}
@@ -6,7 +6,7 @@
 {{- $priority := or .priority "" -}}
 {{- $area := "main" -}}
 
-{{- $managed_keyring := "true" -}}
+{{- $managed_keyring := "false" -}}
 {{- $managed_keyring_repo := "radxa-pkg/radxa-archive-keyring" -}}
 
 {{- $architecture := .architecture -}}

If the checked-out rbuild does not contain managed_keyring option, we need to include the following patch before making changes:

git reset --hard $RBUILD_REVISION
git am --abort
curl -L https://github.com/radxa-repo/rbuild/commit/2a861f6fbc2c1d081d5d83aabfc99bda4abd38d3.patch | git am

We can then use RBUILD_COMMAND as a reference to reproduce the image. The exact command listed in /etc/radxa_image_fingerprint was meant to be run on GitHub's Ubuntu runner as root.

Create apt repo from scratch

To recreate Radxa apt repo from scratch, you will need to fork all related repos.

Even though they can be all forked under the same account, we recommend you create 1 organization specifically to host [radxa-pkg] repos.