Building and booting fully customized OS on Raspberry Pi

Booting Ubuntu desktop on Raspberry pi 3

Motivation

By default, Raspberry pi boots from SD card, and root filesystem is also mounted from there. This is not really good practice, since SD card is not really reliable, and if it’s written frequently, it wears off fast. So at least, I wanted to mount root filesystem from external storage for reliability, maintainability, and performance.

Also, I found that, Raspbian is using 32bit binary, and the kernel is compiled to 32 bit code as well. ARM Cortex-A53 is 64bit processor, and potentially be able to perform 10–15% better if it runs 64bit code (https://www.cnx-software.com/2016/03/01/64-bit-arm-aarch64-instructions-boost-performance-by-15-to-30-compared-to-32-bit-arm-aarch32-instructions/).

Preparation

To build both, custom bootloader and Linux kernel, we need toolchain to generate AARCH64 binary. Toolchain is set of compilers and other tools like linker.

Fortunately, I already had toolchain for ARM processors (both 32bit and 64bit, thanks to Xilinx), so I skipped this step. But this is the most important step for the purpose. For Raspberry pi, toolchain is available like link below.

https://github.com/abhiTronix/raspberry-pi-cross-compilers

Building custom bootloader

One way to do this is, using custom bootloader “u-boot”. u-boot is, as I introduced it in the previous article, a second-stage bootloader, which is capable of booting kernel from various devices. It’s also capable of booting kernel from network, but it’s not the scope of this article.

Raspberry Pi, by default, has 2 bootloaders; bootcode.bin and start.elf, and we need both before booting u-boot (Because Raspberry pi is using very odd SoC, which has GPU in it. Those bootloaders take care of GPU initialization.). Also, it is easier to reuse default “device tree blobs”. So I will reuse “boot” partition for Raspbian, already installed by default (if you don’t have working SD card, prepare it for later use).

Download the source code using git.

git clone git://git.denx.de/u-boot.git

Before building, this has to be configured for the specific target hardware. In my case, I configured it specifically for “Raspberry pi 3B”. You can find the default config for Raspberry pis by doing this;

ls configs | grep rpi

Then, you’ll get output like below;

rpi_0_w_defconfig
rpi_2_defconfig
rpi_3_32b_defconfig
rpi_3_b_plus_defconfig
rpi_3_defconfig
rpi_4_32b_defconfig
rpi_4_defconfig
rpi_arm64_defconfig
rpi_defconfig

For Raspberry pi 3, the right one is “rpi_3_defconfig”. To apply this config, use make command like below (You can export ARCH and CROSS_COMPILE variables).

make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_3_defconfig

You have to replace the default config for your hardware. Also, you can customize “default boot command”, “timeout” and so on, by using interactive “menuconfig”, but it’s not mandatory.

make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- menuconfig

Then build it. You can specify number of parallel jobs by specifying “-j” option.

make ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- -j 4

After building, copy “u-boot” to “boot” partition on SD card, renaming it to “kernel8.img”. The number could be anything higher than “7”, because higher number will be chosen by previous stage bootloader (by default, there are “kernel.img” and “kernel7.img” for compatibility). Since we will be using “u-boot” after default bootloader, we don’t need “config.txt” in boot partition. Rename it to “config.txt.orig” or something to eliminate this from boot process.

Then, you have to prepare “boot script” for u-boot. Here’s an example (rpi_bootscript.txt).

mmc dev 0                                                                   
fatload mmc 0:1 ${kernel_addr_r} Image                                      
fatload mmc 0:1 ${fdt_addr_r} bcm2710-rpi-3-b.dtb                           
setenv bootargs "console=serial0,115200 console=tty1 root=/dev/sda1 rw rootfs type=ext4 rootwait devtmpfs.mount=0"                                       
booti ${kernel_addr_r} - ${fdt_addr_r}

Basically, what it does is, reading “kernel image”, “device tree blob” and (if needed) “initrd” from SD card, and put it into DRAM. Also, the important thing is “kernel bootarg”. It is important to customize this since we are mounting root filesystem from external storage (root=/dev/sda1).

Create image for boot script. “mkimage” is in <u-boot>/tools. Add the path before making boot script.

export PATH=${PATH}:tools # Note, "tools" is in u-boot directory
mkimage -A arm -O linux -T script -a 0 -e 0 -C none -d rpi_bootscript.txt boot.scr

Building Linux kernel

Download the kernel source code.

git clone --depth=1 https://github.com/raspberrypi/linux

By default, kernel version is 4.19, which is pretty old. If you want to use newer kernel, checkout newer branch. I checked out “rpi-5.3.y”, since, this PC’s kernel version is 5.3, but again, this is not mandatory.

git checkout rpi-5.3.y

Configure the kernel, and build it like u-boot. For kernel, config files are in “arch/arm64/configs” folder.

ls arch/arm64/configs

This time, there are only 3 config files. If it is Raspberry pi 3, choose bcmrpi3_defconfig.

make ARCH=arm64 CROSS_COMIPLE=aarch64-linux-gnu- bcmrpi3_defconfig

Do menu config if necessary.

make ARCH=arm64 CROSS_COMIPLE=aarch64-linux-gnu- menuconfig

Build the kernel (it takes a while). Choose right number of jobs to reduce the compile time.

make ARCH=arm64 CROSS_COMIPLE=aarch64-linux-gnu- Image modules -j4

When it’s built, you need to copy the kernel, and install modules. Module installation will be later since we need to prepare root filesystem first to install those. Copy kernel (arch/arm64/boot/Image) to your boot partition on SD card.

Preparing root filesystem

Here’s what I referred before doing it by myself. But not everything written here worked fine for me, so my procedure is slightly different.

https://a-delacruz.github.io/ubuntu/rpi3-setup-filesystem.html

Download the tar archive using wget.

wget http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.1-base-arm64.tar.gz

Create “rootfs” directory anywhere, and extract root filesystem in it.

sudo tar xzvf ubuntu-base-18.04.1-base-arm64.tar.gz -C <path to the root>/rootfs

Because we are setting up root filesystem for “AARCH64”, we need emulator to execute commands on foreign filesystem. Install “qemu”.

sudo apt install qemu-user-static

Then copy this binary into root filesystem.

sudo cp /usr/bin/qemu-aarch64-static rootfs/usr/bin/

Also, to run “apt” command, you need “resolv.conf” from host PC.

sudo cp /etc/resolv.conf rootfs/etc/

Now, it’s time to install Linux kernel modules. Go back to the linux source directory, and run this command.

sudo make ARCH=arm64 modules_install INSTALL_MOD_PATH=<path to root>/rootfs

From here, you can use “chroot” command, and setup this root filesystem as if it is running on host PC.

sudo chroot <path to root>/rootfs

The setup procedure will be like this;

1. Create user and set password
2. Set host name
3. Update
4. Setup locale (language pack)
5. Install necessary modules.
6. Change fstab
7. Create partition on external storage (USB SSD)
8. Copy root filesystem

First, create user and set password.

useradd -G sudo -m -s /bin/bash <user name>
passwd <user name>

Set password for root as well.

passwd root

Change host name.

echo “<host name>” > /etc/hostname
echo "127.0.0.1	localhost" > /etc/hosts
echo "127.0.1.1	<host name>" >> /etc/hosts

Then update.

apt update

Before doing “upgrade”, we had better setup language pack since it will warn us there’s no language pack installed.

apt install locales
locale-gen “en_US.UTF-8”

In my case, I added “locale-gen en_CA.UTF-8” as well. Then, upgrade.

apt upgrade

I would recommend installing modules one by one, since I had bad time installing it at once. Also, I would recommend installing modules in this order (especially, ssh caused me a lot of trouble). But in this example, I put everything in one line.

apt install sudo apt-utils dialog ssh kmod udev perl net-tools ethtool ifupdown dhcpcd5 wireless-tools iputils-ping wpasupplicant vim

Then, add this to fstab to avoid mounting root filesystem as “read-only”.

echo "/dev/sda1	/	ext4	defaults,noatime	0	1" >> <path to root>/rootfs/etc/fstab

It would be better if you specify “UUID” rather than device path “/dev/sda1”.

Then, return to host PC.

exit

Plug the USB external SSD. Make sure, which device you are installing, otherwise, you will overwrite your host PC’s storage!! You can type “lsblk” or something similar to make sure, which is the right device to install.

Create the partition.

sudo fdisk /dev/sda

In my case, the new device was “sda”, but please double check.

Then, type “d” if there are any partitions already existing. If not, create new partition by pressing “n”. You can use default values for this new partition creation. “w” to write changes (If you get error here, make sure this device is not mounted anywhere, and try it again).

Create file system.

sudo mkfs.ext4 -L root /dev/sda1

Then, you can copy everything you prepared using chroot.

sudo mount /dev/sda1 /mnt/tmp
sudo rsync -av --progress <path to root>/rootfs /mnt/tmp

Now, everything is ready. Unmount the drive, and plug it to Raspberry pi. Also, put the SD card, and turn on the device.

That’s all!!

If you need GUI, you may want to install lxde. I don’t recommend ubuntu-desktop since it eats up all the memory, and no other applications could launch.