Build Your ARM Image for QEMU

Hello there! I’ve been using QEMU for a while. Recently, I updated my ARM image and found that it has some difficulties to make an ARM image and could be an impossible job for those who are not familiar with embedded systems. I decided to write a comprehensive post to share some knowledge and how to build you own ARM image. This post could be very useful for those who want to know how Linux works with minimum requirements (root file system, in short rootfs). I use Vexpress as the target board in the tutorial.

This post is separated into three parts. The first part teaches you how a Linux works from a programmer’s view who wants to port Linux on different platforms, such as QEMU. The second part is a tutorial for building a Linux kernel and rootfs running on QEMU. The last part is how to run existing kernel and images on QEMU. I believe this is very comprehensive for someone who want to learn or start a new project on ARM with QEMU or other ARM platforms.

How Does a Linux Boot

Many people know boot loader and some Linux boot sequence. You can Google it to find the detail version. Here I would like to point out how QEMU boot Linux, which is the most central and fundamental part that one should understand!!

QEMU has a command argument called “-kernel”. It is a very handy function!! Because of this feature, we don’t need to bother the complicated boot sequence and problems on locating Kernel Image. We could just use “-kernel” to pass kernel image to QEMU. Then, QEMU will uncompress the kernel image to a proper memory location and start to run the kernel code. This function saves the time to figure out how to boot Kernel on QEMU.

Now, let’s talk about how Linux Boot. If you read the paragraph above, then you are done!! All you need to do to run a kernel is to specify kernel image with “-kernel” argument to QEMU!!  Linux Kernel is a complete program that could be run without other shared libraries. So… that’s it!! However!!! it is useless to run it that way because the only thing you can do is watching it printing messages which end up with “kernel panic …. unable to mount rootfs …”. This indicates that kernel could not complete “init” procedure. So, next section will teach you how to build a rootfs (root file system) to run a Linux Kernel correctly.

Finally, I would like to talk about kernel modules. Of cause, you can build all modules in your kernel, so that you don’t need to deal with this problem. Since this is a tutorial, I want to dive deeper. Kernel modules are mostly drivers, both hardware drivers and software drivers.  For example, the Ethernet!  If the driver is a kernel module stored in root file system, Linux kernel will not be able to access the Internet before mounting the root file system. Another example is ext3, ext4 driver, Linux Kernel must contain these basic file system driver in order to execute init procedure because the init files are located in root file system. It’s somehow a very common problem which was very popular in early years. That’s why we have so-called initramfs or rootfs. They are minimal file system images containing all kernel modules(.ko files), init procedure scripts, and necessary binaries to boot a full system. In the following tutorial, we will build the ethernet driver to a separate .ko file to illustrate the usage of a kernel module on ethernet.

Last part about booting Linux is device tree. A device tree is a tree data structure with nodes that describe the physical devices in a system. Linux reads the dtb(device tree blob) file to know the underlying hardware information and achieves portability. To further understand what information are included in the device tree, you can read the following articles.

1. Wiki
2. Device Tree for Dummies
3. Xilinx Zynq Example

How to Build an ARM Image

To build ARM image running on QEMU. One should prepare the following things. These are all the repos we are going to cover in this section. To compile QEMU correctly, you might need to install some packages which are not mentioned in this article.

Git Repos:

• QEMU: https://github.com/qemu/qemu
• Linux: https://github.com/torvalds/linux
• Busybox: (Deprecated, use the link below)
• Buildroot: git://git.buildroot.net/buildroot

Websites:

• Busybox: https://busybox.net/downloads/
• Linaro ARM compiler(hard floating): gcc-linaro-4.9-2014.11-x86_64_arm-linux-gnueabihf.tar.xz
• Linaro ARM compiler: gcc-linaro-4.9-2014.11-x86_64_arm-linux-gnueabi.tar.xz

Prepare ARM compiler:

First, download Linaro ARM compiler with “hf” (hard floating point calling convention) support!! You can download it from the link above. Extract it and put it in a proper location. Then, edit your .bashrc file and export an environment variable like this. Here I put it to “~/bin/linaro-arm-linux-gnueabihf”. You need to modify this path to your location.

NOTE: If you are doing this for other platforms without hardware floating points supports, use the one without “hf” instead. The following commands and scripts are written with linaro-arm-linux-gnueabihf compiler.

export PATH=~/bin/linaro-arm-linux-gnueabihf/bin:$PATH
export PATH=~/bin/linaro-arm-linux-gnueabi/bin:$PATH

Then, reload the file by:

~$source ~/.bashrc

Compiling Linux Kernel:

Here, I used Linux 4.4 as my image version. You may or may not need some adjustments in newer versions. Download and compile Linux kernel with target board default configuration. There are two configurations you need to modify. First is “CONFIG_FHANDLE“, please refer to thread1, thread2 for more information about why doing this. Second is the ethernet driver SMC91C1xxx.

Enable CONFIG_FHANDLE by pressing <Y> key.

General setup --->
[*] open by fhandle syscalls

Set devtmpfs settings for correct behavior of /dev system. If this feature is not set, some device driver might not work and report “Internal Error” or “Major Minor number not found” errors due to the major-minor number would be redirected to an incorrect number when this feature is disabled.

Device Drivers  --->
  Generic Driver Options  --->
    [*] Maintain a devtmpfs filesystem to mount at /dev
    [*]   Automount devtmpfs at /dev, after the kernel mounted the rootfs

Set SMC91C1xxx to loadable kernel module by pressing <M> key.

P.S. This  is just for tutorial use. For convenience, you don’t need to make this change in the real situation. Just leave it included, shown as <*>. If you mark it included, you might probably don’t do anything with root file system since kernel modules are already included in your kernel image. If you mark it loadable, you will need to copy the kernel modules into root file system to make it work correctly.

Device Drivers ---> Network device support ---> Ethernet driver support --->
[*]   SMC (SMSC)/Western Digital devices
<M>     SMC 91C9x/91C1xxx support
< >     SMSC LAN911[5678] support
<M>     SMSC LAN911x/LAN921x families embedded ethernet support

# Download linux
~$git clone https://github.com/torvalds/linux
~$cd linux
# Switch to version 4.4 (It’s not necessary!!!!)
~/linux$git checkout v4.4
# Load default config for target board
~/linux$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make vexpress_defconfig
# Adjust some settings as mentioned above
~/linux$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make menuconfig
# Compile the kernel
~/linux$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j4

 

After successfully compiling Linux, the image we need is located in “arch/arm/boot/zImage” and device tree file for Vexpress is in “arch/arm/boot/dts/vexpress-v2p-ca9.dtb”. That’s it! You can now run it with QEMU to get a first taste by running:

~/linux$qemu-system-arm \
-M vexpress-a9 \
-dtb ./arch/arm/boot/dts/vexpress-v2p-ca9.dtb \
-kernel ./arch/arm/boot/zImage \
-append "console=ttyAMA0" \
--nographic

 

You will stop with a kernel panic, but it’s fine. We still lack root file system for Linux kernel to complete the init procedure. Next, we are going to build a root file system for Linux to run.

Please append a correct "root=" boot option; here are the available partitions:
1f00          131072 mtdblock0  (driver?)
1f01           32768 mtdblock1  (driver?)
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.4.0+ #5
Hardware name: ARM-Versatile Express
[<8001640c>] (unwind_backtrace) from [<80012f28>] (show_stack+0x10/0x14)
[<80012f28>] (show_stack) from [<8025978c>] (dump_stack+0x88/0x98)
[<8025978c>] (dump_stack) from [<800a4cd0>] (panic+0xa0/0x204)
[<800a4cd0>] (panic) from [<80646254>] (mount_block_root+0x1c0/0x25c)
[<80646254>] (mount_block_root) from [<8064640c>] (mount_root+0x11c/0x124)
[<8064640c>] (mount_root) from [<8064656c>] (prepare_namespace+0x158/0x19c)
[<8064656c>] (prepare_namespace) from [<80645ef0>] (kernel_init_freeable+0x268/0x278)
[<80645ef0>] (kernel_init_freeable) from [<804b8f44>] (kernel_init+0xc/0xe8)
[<804b8f44>] (kernel_init) from [<8000f538>] (ret_from_fork+0x14/0x3c)

 

Root File System (rootfs)

There are two ways to make a root file system, Busybox and Buildroot. Busybox is a very old way to build a minimal root file system with only basic binaries. The user needs to create “/dev” and “/etc” folders with appropriate settings. On the other hand, Buildroot, integrating many pieces of stuff, is a very easy way with lots of supports of different libraries and binaries to build a root file system under your needs. Personally, I prefer Buildroot since build root is more easy to add some extra binaries, libraries and system services(systemV, systemd, etc.).

Download Busybox and set the setting as follows:

Busybox Settings ---> Build Options --->
[*] Build BusyBox as a static binary (no shared libs)

Instructions:

~$cd busybox
# Adjust some settings as mentioned above
~/busybox$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make menuconfig
# Compile Busybox
~/busybox$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j4

Next, you need to create basic device nodes and init scripts. I prepare the scripts to complete this task and make life easier. The origin is from this post; I’m not the person who wrote this script, but I modified it.

#!/bin/bash

#Remove old files
sudo rm -rf rootfs
rmdir tmpfs
rm -f a9rootfs.ext3

#Create new rootfs folder
sudo mkdir rootfs
sudo cp busybox/_install/* rootfs/ -raf

#Create folders required by Linux convention
sudo mkdir -p rootfs/proc/
sudo mkdir -p rootfs/sys/
sudo mkdir -p rootfs/tmp/
sudo mkdir -p rootfs/root/
sudo mkdir -p rootfs/var/
sudo mkdir -p rootfs/mnt/

#Download sample etc files with init procedure and copy it to rootfs
wget http://files.cnblogs.com/files/pengdonglin137/etc.tar.gz
tar -xf ./etc.tar.gz
sudo cp etc rootfs/ -arf

#Copy shared libraries to rootfs
sudo cp -arf ~/bin/linaro-arm-linux-gnueabihf/libc/usr/lib rootfs/
sudo rm rootfs/lib/*.a
sudo arm-linux-gnueabihf-strip rootfs/lib/*

#Create basic device nodes
sudo mkdir -p rootfs/dev/
sudo mknod rootfs/dev/tty1 c 4 1
sudo mknod rootfs/dev/tty2 c 4 2
sudo mknod rootfs/dev/tty3 c 4 3
sudo mknod rootfs/dev/tty4 c 4 4
sudo mknod rootfs/dev/console c 5 1
sudo mknod rootfs/dev/ttyAMA0 c 204 64
sudo mknod rootfs/dev/null c 1 3

#Create ext3 image file
dd if=/dev/zero of=a9rootfs.ext3 bs=1M count=$((32))
mkfs.ext3 a9rootfs.ext3

#Copy all the files in our rootfs to image
mkdir -p tmpfs
sudo mount -t ext3 a9rootfs.ext3 tmpfs/ -o loop
sudo cp -r rootfs/* tmpfs/
sudo umount tmpfs
rmdir tmpfs

Now, you can run QEMU with this rootfs.

~/$qemu-system-arm \
-M vexpress-a9 \
-dtb ./linux/arch/arm/boot/dts/vexpress-v2p-ca9.dtb \
-kernel ./linux/arch/arm/boot/zImage \
-append "root=/dev/mmcblk0 console=ttyAMA0" \
-sd ./a9rootfs.ext3 \
--nographic

 

You will see the following messages when you succeed.

mmc0: new SD card at address 4567
mmcblk0: mmc0:4567 QEMU! 32.0 MiB
input: AT Raw Set 2 keyboard as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10006000.kmi/serio0/input/input0
aaci-pl041 10004000.aaci: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
aaci-pl041 10004000.aaci: FIFO 512 entries
oprofile: using arm/armv7-ca9
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
Registering SWP/SWPB emulation handler
Key type encrypted registered
rtc-pl031 10017000.rtc: setting system clock to 2016-03-29 08:30:44 UTC (1459240244)
ALSA device list:
#0: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 33
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/smb/smb:motherboard/smb:motherboard:iofpga@7,00000000/10007000.kmi/serio1/input/input2
EXT4-fs (mmcblk0): mounting ext3 file system using the ext4 subsystem
EXT4-fs (mmcblk0): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext3 filesystem) readonly on device 179:0.
Freeing unused kernel memory: 292K (80645000 - 8068e000)
random: nonblocking pool is initialized
mkdir: can't create directory '/var/lock': Read-only file system

Please press Enter to activate this console.

The kernel successfully catches our SD card device with the name “QEMU” and the size 32MB, mapping at address 4567.  After pressing Enter, you will login single user mode, which has only root account to do some maintenance. It is not convenient and sometimes causes problems on executing a binary compiled by yourself. It’s just a toy that shows some functionality of Linux kernel but without user applications. So, I strongly suggest using Buildroot to generate a minimal rootfs.

Buildroot, Making Embedded Linux Easy

Busybox does not complete the full system we need. In addition, it is very complicated for developers to understand the whole Linux root filesystem and init procedures. Buildroot is the perfect solution which includes many repositories, binaries, libraries from elsewhere. Buildroot is a tool providing a menuconfig UI. Developers can quickly pick the system configurations they want and choose libraries, binaries to be installed in the rootfs. The only drawback of using buildroot is the download and compilation time since all the source codes, and even compilers are downloaded by its scripts to ensure the integrity and functionality. Trading the compilation time, developers save the time to build the environment and find the source codes of those libraries. I think it is very worthy.

To download(clone) and configure buildroot for our needs is super easy.

~$git clone git://git.buildroot.net/buildroot
~$cd buildroot
# Please configure the settings listed below
~/buildroot$make menuconfig
# Don’t use -j
~/buildroot$make

Then, configure the basic, minimal rootfs as the following settings:

#Set to the proper processor if you use different CPU
Target options --->
    Target Architecture (ARM (little endian))
    Target Binary Format (ELF)
    Target Architecture Variant (cortex-A9)
    Target ABI (EABI)
    Floating point strategy (Soft float)
    ARM instruction set (ARM)
#I prefer this toolchain rather than buildroot's
Toolchain --->
    Toolchain type (External toolchain)
    Toolchain (Sourcery CodeBench ARM 2014.05)
    Toolchain origin (Toolchain to be downloaded and installed)
#This is the minimal setting one could use. 
System configuration --->
    Init system (BusyBox)
    /dev management (Dynamic using devtmpfs only)
    [*] Enable root login with password
    (root) Root password
        /bin/sh (busybox' default shell)
#CPIO is more compatible and has readonly property when executing QEMU
Filesystem images --->
    [*] cpio the root filesystem (for use as an initial RAM filesystem)

Don’t forget to set a password for root user here if you choose “Enable root login with password” as I do here. This sample setting is the basic setup I prefer to use. Actually, one could just set “Target options” to proper ARM CPU and “make”. Here, my setting is trying to demonstrate some critical rootfs settings one might need to choose.
If you prefer fancy terminal instead of basic stuff, you can choose different init system. Like this:

#This example uses different init system and shell
System configuration --->
    (hello_rootfs) System hostname
    (From Medicine's Blog) System banner
    #You can try systemd if you want :P
    Init system (systemV)
        #This is the default shell after login.
        /bin/sh (bash)
#Here, you can install multiple shells
Target packages ---> Shell and utilities --->
    -*- bash
    [*] zsh

There are many convenient binaries you can install. Ex: sudo, htop, tmux, vim, python, PHP, etc. and libraries, such as MySQL, zlib, libfuse, Javascript, x264, x265, etc. Exploring how many binaries and libraries we could install are very interesting! Take some time to look at all options! You might find something useful. Here I set default shell to bash for more powerful and complete functions. You can pick zsh or other shells you like as default.

After a successful compilation, your rootfs.cpio file will be located in “buildroot/output/images/rootfs.cpio”

You can use the kernel compiled previously with this new image.

~/$qemu-system-arm \
-M vexpress-a9 \
-dtb ./linux/arch/arm/boot/dts/vexpress-v2p-ca9.dtb \
-kernel ./linux/arch/arm/boot/zImage \
-append "console=ttyAMA0" \
-initrd ./buildroot/output/images/rootfs.cpio \
--nographic

 

INIT: version 2.88 booting
Starting logging: OK
Initializing random number generator...
done.
Starting network...
INIT: Entering runlevel: 3

From Medicine's Blog
hello_rootfs login:

This message means you made it!! Now, it’s running multi-user mode under systemV init procedure. You can log in “root” with password “root”. If you fail to login, you might forget to set the password for root when configuring Buildroot.

That’s it!! Enjoy your easy life with Buildroot. 😀

Arch Linux (ARM)

Now, let’s move to running a more rich and powerful Linux system on ARM QEMU. First, go to ArchLinux|ARM website and download latest image of Vexpress. In case you can’t find it, here’s the link. The commands to extract the file and build a root file system are very simple. Basically, what one needs to do is extract it in a folder. But what we are going to do is to build a rootfs image for QEMU, we need to create an empty ext4 image and put everything in it.

~/$mkdir root
~/$dd if=/dev/zero of=./arch_rootfs.ext4 bs=1M count=$((4 * 1024))
~/$mkfs.ext4 ./arch_rootfs.ext4
~/$sudo mount ./arch_rootfs.ext4 ./root
~/$wget http://archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz
~/$sudo bsdtar -xpf ArchLinuxARM-armv7-latest.tar.gz -C ./root
~/$sync
~/$cp ./root/boot/zImage ./zImage_arch
~/$cp ./root/boot/dtbs/vexpress-v2p-ca9.dtb ./
~/$sudo umount ./root

Now, the linux Kernel and Arch rootfs are ready. We don’t need boot loader here. You can now execute QEMU with the following command:

~/$qemu-system-arm \
-M vexpress-a9 \
-dtb ./vexpress-v2p-ca9.dtb \
-kernel ./zImage_arch \
-append "root=/dev/mmcblk0 rw roottype=ext4 console=ttyAMA0" \
-drive if=sd,driver=raw,cache=writeback,file=./arch_rootfs.ext4 \
--nographic \
-snapshot

The message should be like this. You can log in “root” with password “root”.

......
Welcome to Arch Linux ARM!
[  OK  ] Reached target Paths.
[    6.270852] systemd[1]: Listening on udev Control Socket.
[  OK  ] Listening on udev Control Socket.

......
Arch Linux 4.4.3-1-ARCH (ttyAMA0)

vexpress-ARM login:

I use “-snapshot” to make all the changes in the image volatile for safety. If I run QEMU without “-snapshot”, all the changes I made on QEMU VM would be persistent. However, if I terminate QEMU with the signal “ctrl+a then press c”, some files in the ext4 rootfs image might be broken. To prevent this tragedy, I use “-snapshot” to drop all the writes and changes. Of course, if one shutdown QEMU with “poweroff” command on guest OS, everything will be fine.

Running Arch Linux With Your Kernel

While running Arch with precompiled Linux kernel is easy, running Arch your kernel might be hard. There are three steps in total. Copy the Linux kernel config file from rootfs. Compile Linux with that “.config” file and compile kernel modules. Install your kernel modules on rootfs. That’s it! It’s really easy but a little bit complicated when you do it.

For those who are familiar with kernel compile, I prepare a script to do all the jobs and make our life easier. If you are familiar with the process of compiling kernel and kernel modules, you can do it by yourself as well. It’s the same thing when you compile your own kernel and kernel modules on your x86 systems. Just don’t forget to put kernel headers and kernel modules in a correct location in rootfs.

To retrieve the kernel config, you need to launch the image with QEMU. After you launch QEMU, all you need to do is zcat.

# In QEMU Guest OS
~$zcat /proc/config.gz

Now, you can compile your own kernel. There’s one thing you must do is removing all the external firmware drivers (CONFIG_EXTRA_FIRMWARE=””). They are not open source and hard to download. Unless you need to run linux on thoes developement boards, just remove them.

# Make sure this field is empty
Device Drivers ---> Generic Driver Options --->
    ()    External firmware blobs to build into the kernel binary

# Download linux
~$git clone https://github.com/torvalds/linux
~$cd linux
# Switch to the version of your Arch ARM
~/linux$git checkout v4.4
# Load the external config we extracted from Arch ARM
~/linux$cp ../arch_arm_config ./.config
# Remove all external firmware drivers which we don’t have
# And adjust some settings if you need
~/linux$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make menuconfig
# Compile the kernel
~/linux$ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make -j4
# Package the files we need. Need about 5 mins
~/linux$../install.sh
# Mount Arch ARM rootfs image
~/linux$cd ../
~$sudo mount ./arch_rootfs.ext4 ./root
~$cd linux
# Copy packed kernel modules and headers
~/linux$sudo cp -r ./output/usr/lib/modules/* ../root/usr/lib/modules/
# Copy zImage to where you place kernel image
~/linux$cp ./output/boot/zImage ../zImage_arch
# Un-mount Arch ARM rootfs image
~/linux$cd ../
~$sudo umount ./root

Here’s the script, please copy paste it manually. The variables from line 4 to line 15 are related to kernel version and naming. Just modify the version number to your Linux kernel version number. After executing this script, the output files are in “./output” folder. Please copy all the files in “./output” to your rootfs and copy “./output/boot/zImage” to where you place Linux kernel image.

#!/bin/bash
buildarch=4

KARCH=arm
pkgbase=linux-armv7
_srcname=linux-4.4
_kernelname=${pkgbase#linux}
_desc="ARMv7 multi-platform"
pkgver=4.4.0
pkgrel='ARCH'
_kernver="${pkgver}-${pkgrel}"
arch='armv7h'
pkgdir=./output
_basekernel=${_kernver%%-*}
_basekernel=${_basekernel%.*}

function generate_headers() {
  install -dm755 "${pkgdir}/usr/lib/modules/${_kernver}"

  install -D -m644 Makefile \
    "${pkgdir}/usr/lib/modules/${_kernver}/build/Makefile"
  install -D -m644 kernel/Makefile \
    "${pkgdir}/usr/lib/modules/${_kernver}/build/kernel/Makefile"
  install -D -m644 .config \
    "${pkgdir}/usr/lib/modules/${_kernver}/build/.config"

  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/include"

  for i in acpi asm-generic config crypto drm generated keys linux math-emu \
    media net pcmcia scsi sound trace uapi video xen; do
    cp -a include/${i} "${pkgdir}/usr/lib/modules/${_kernver}/build/include/"
  done

  # copy arch includes for external modules
  mkdir -p ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH
  cp -a arch/$KARCH/include ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH/
  for i in dove exynos mvebu omap2 versatile; do
    mkdir -p ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH/mach-${i}
    cp -a arch/$KARCH/mach-${i}/include ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH/mach-${i}/
  done
  for i in omap orion samsung versatile; do
    mkdir -p ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH/plat-${i}
    cp -a arch/$KARCH/plat-${i}/include ${pkgdir}/usr/lib/modules/${_kernver}/build/arch/$KARCH/plat-${i}/
  done

  # copy files necessary for later builds, like nvidia and vmware
  cp Module.symvers "${pkgdir}/usr/lib/modules/${_kernver}/build"
  cp -a scripts "${pkgdir}/usr/lib/modules/${_kernver}/build"

  # fix permissions on scripts dir
  chmod og-w -R "${pkgdir}/usr/lib/modules/${_kernver}/build/scripts"
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/.tmp_versions"

  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/arch/${KARCH}/kernel"

  cp arch/${KARCH}/Makefile "${pkgdir}/usr/lib/modules/${_kernver}/build/arch/${KARCH}/"

  cp arch/${KARCH}/kernel/asm-offsets.s "${pkgdir}/usr/lib/modules/${_kernver}/build/arch/${KARCH}/kernel/"

  # add docbook makefile
  install -D -m644 Documentation/DocBook/Makefile \
    "${pkgdir}/usr/lib/modules/${_kernver}/build/Documentation/DocBook/Makefile"

  # add dm headers
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/md"
  cp drivers/md/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/md"

  # add inotify.h
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/include/linux"
  cp include/linux/inotify.h "${pkgdir}/usr/lib/modules/${_kernver}/build/include/linux/"

  # add wireless headers
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/net/mac80211/"
  cp net/mac80211/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/net/mac80211/"

  # add dvb headers for external modules
  # in reference to:
  # http://bugs.archlinux.org/task/9912
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-core"
  cp drivers/media/dvb-core/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-core/"
  # and...
  # http://bugs.archlinux.org/task/11194
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/include/config/dvb/"
  cp include/config/dvb/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/include/config/dvb/"

  # add dvb headers for http://mcentral.de/hg/~mrec/em28xx-new
  # in reference to:
  # http://bugs.archlinux.org/task/13146
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-frontends/"
  cp drivers/media/dvb-frontends/lgdt330x.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-frontends/"
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/i2c/"
  cp drivers/media/i2c/msp3400-driver.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/i2c/"

  # add dvb headers
  # in reference to:
  # http://bugs.archlinux.org/task/20402
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/usb/dvb-usb"
  cp drivers/media/usb/dvb-usb/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/usb/dvb-usb/"
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-frontends"
  cp drivers/media/dvb-frontends/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/dvb-frontends/"
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/tuners"
  cp drivers/media/tuners/*.h "${pkgdir}/usr/lib/modules/${_kernver}/build/drivers/media/tuners/"

  # add xfs and shmem for aufs building
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/fs/xfs"
  mkdir -p "${pkgdir}/usr/lib/modules/${_kernver}/build/mm"

  # copy in Kconfig files
  for i in $(find . -name "Kconfig*"); do
    mkdir -p "${pkgdir}"/usr/lib/modules/${_kernver}/build/`echo ${i} | sed 's|/Kconfig.*||'`
    cp ${i} "${pkgdir}/usr/lib/modules/${_kernver}/build/${i}"
  done

  #chown -R root.root "${pkgdir}/usr/lib/modules/${_kernver}/build"
  find "${pkgdir}/usr/lib/modules/${_kernver}/build" -type d -exec chmod 755 {} \;

  # strip scripts directory
  find "${pkgdir}/usr/lib/modules/${_kernver}/build/scripts" -type f -perm -u+w 2>/dev/null | while read binary ; do
    case "$(file -bi "${binary}")" in
      *application/x-sharedlib*) # Libraries (.so)
        /usr/bin/strip ${STRIP_SHARED} "${binary}";;
      *application/x-archive*) # Libraries (.a)
        /usr/bin/strip ${STRIP_STATIC} "${binary}";;
      *application/x-executable*) # Binaries
        /usr/bin/strip ${STRIP_BINARIES} "${binary}";;
    esac
  done

  # remove unneeded architectures
  rm -rf "${pkgdir}"/usr/lib/modules/${_kernver}/build/arch/{alpha,arc,arm26,arm64,avr32,blackfin,c6x,cris,frv,h8300,hexagon,ia64,m32r,m68k,m68knommu,metag,mips,microblaze,mn10300,openrisc,parisc,powerpc,ppc,s390,score,sh,sh64,sparc,sparc64,tile,unicore32,um,v850,x86,xtensa}

}

function generate_modules() {
  mkdir -p "${pkgdir}"/{lib/modules,lib/firmware}
  ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make INSTALL_MOD_PATH="${pkgdir}" modules_install
  ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- make INSTALL_DTBS_PATH="${pkgdir}/boot/dtbs" dtbs_install
  cp arch/$KARCH/boot/zImage "${pkgdir}/boot/zImage"

  # remove build and source links
  rm -f "${pkgdir}"/lib/modules/${_kernver}/{source,build}
  # remove the firmware
  rm -rf "${pkgdir}/lib/firmware"
  # gzip -9 all modules to save 100MB of space
  find "${pkgdir}" -name '*.ko' |xargs -P 2 -n 1 gzip -9
  # make room for external modules
  ln -s "../extramodules-${_basekernel}-${_kernelname:-ARCH}" "${pkgdir}/lib/modules/${_kernver}/extramodules"
  # add real version for building modules and running depmod from post_install/upgrade
  mkdir -p "${pkgdir}/lib/modules/extramodules-${_basekernel}-${_kernelname:-ARCH}"
  echo "${_kernver}" > "${pkgdir}/lib/modules/extramodules-${_basekernel}-${_kernelname:-ARCH}/version"

  # Now we call depmod...
  depmod -b "$pkgdir" -F System.map "$_kernver"

  # move module tree /lib -> /usr/lib
  cp -r "$pkgdir/lib" "$pkgdir/usr"
  rm -rf "$pkgdir/lib"
}

generate_headers
generate_modules

At the end, I would like to comment something you need to do for a better experience of running Arch ARM on QEMU.

First, you might need to compile Linux kernel and enable “open by fhandle syscalls” as mentioned in this tutorial. Otherwise, you might wait for 1’30 for ttyAMA0. Reference: thread1, thread2.

Second, there are many useless system services and unworkable services. You might be interested in clean up systemd. After launching Arch ARM on QEMU and logging in, you can mask out all the unworkable(failed) services in your Arch ARM. After doing this, don’t forget to gently “poweroff” Arch ARM.

Choose of Linaro Compiler

I would like to comment one of the most important things in developing the application and the root file system. The Compiler. There are many toolchain providers on the web, but there is one very popular and continuously updating their toolchain, Linaro. Linaro also provides Linux kernels for ARM. I didn’t choose that because I’m more familiar with mainline Linux Kernel. However, Linaro’s kernel is a good choice with security as well.

When you go to the release page of the toolchain, you will see different naming of compilers. If you are building a rootfs for ARM processor supporting hardware floating point unit, feel free to choose the one with “hf” suffix. If your processor does not have a hardware floating point unit, go for the one without “hf”, otherwise, your codes might not work due to unsupported instructions. So, generally, you can use arm-linux-gnueabi for your processor.

arm-linux-gnueabi is the cross-toolchain package for the armel architecture. This toolchain implies the EABI generated by gcc’s -mfloat-abi=softfp options.

arm-linux-gnueabihf is the cross-toolchain package for the armhf architecture. This toolchain implies the EABI generated by the gcc -mfloat-abi=hard option.

Specifying “soft” causes GCC to generate output containing library calls for floating-point operations. “softfp” allows the generation of code using hardware floating-point instructions, but still uses the soft-float calling conventions. “hard” allows generation of floating-point instructions and uses FPU-specific calling conventions.

• arm-linux-gnueabi: The general tool chain with Linux dependency.
• arm-linux-gnueabihf: Utilize hardware floating point registers.
• armeb-linux-gnueabi: Big Endian.
• armeb-linux-gnueabihf: Big Endian. Utilize hardware floating point registers.
• arm-none-eabi: Bare metal. The general tool chain that does not rely on any Linux Kernel.
• armeb-none-eabi: Bare metal. Big Endian.

When you are compiling the Busybox and get an error with “libm.a” which is “-lm” in the args, just don’t use static linking. If that doesn’t help, download arm-linux-gnueabihf. Because of unknown reason, the toolchain with “hf” suffix contains libm.a while the other one does not.

If you want to use Linaro’s tool chain compiling Buildroot. You need to set the following options:

Toolchain --->
    Toolchain (Custom toolchain)
    Toolchain origin (Pre-installed toolchain)
    (/PATH/TO/TOOL_CHAIN/linaro-arm-linux-gnueabihf) Toolchain path
    ($(ARCH)-linux-gnueabi) Toolchain prefix
        External toolchain gcc version (4.9.x)
        External toolchain kernel headers series (3.17.x)
    [*] Toolchain has C++ support?

It seems like Linaro’s toolchain does not pass the eabi test of Buildroot, but we know it supports. We need to remove/delete two lines of the script in order to successfully compile the files. They are two “exit 1” codes at line number 360 and 367 as shown below. The absolute position might be changed.

#
# Check that the Buildroot configuration of the ABI matches the
# configuration of the external toolchain.
#
# $1: cross-gcc path
# $2: cross-readelf path
#
check_arm_abi = \
	__CROSS_CC=$(strip $1) ; \
	__CROSS_READELF=$(strip $2) ; \
	EXT_TOOLCHAIN_TARGET=`LANG=C $${__CROSS_CC} -v 2>&1 | grep ^Target | cut -f2 -d ' '` ; \
	if ! echo $${EXT_TOOLCHAIN_TARGET} | grep -qE 'eabi(hf)?$$' ; then \
		echo "External toolchain uses the unsuported OABI" ; \
		exit 1 \
	fi ; \
	if ! echo 'int main(void) {}' | $${__CROSS_CC} -x c -o $(BUILD_DIR)/.br-toolchain-test.tmp - ; then \
		rm -f $(BUILD_DIR)/.br-toolchain-test.tmp*; \
		abistr_$(BR2_ARM_EABI)='EABI'; \
		abistr_$(BR2_ARM_EABIHF)='EABIhf'; \
		echo "Incorrect ABI setting: $${abistr_y} selected, but toolchain is incompatible"; \
		exit 1 \
	fi ; \
	rm -f $(BUILD_DIR)/.br-toolchain-test.tmp*

At the end, we need to strip out useless debugging information to reduce the size. I choose to keep symbol table for back tracing and function tracing function of my personal study. P.S. You can strip out all unnecessary things if you choose “strip” instead of “none” here. It’s the default action.

~/$arm-linux-gnueabihf-strip --strip-debug --strip-dwo ./rootfs/lib/*
~/$arm-linux-gnueabihf-strip --strip-debug --strip-dwo ./rootfs/usr/lib/*

 

QEMU Arguments

QEMU has very complicated arguments. Here is an example to wrap up all the arguments I used in this tutorial.

• -M: Specify the machine type. Use “-M help” to list all the supported boards
• -kernel: Specify the kernel image (bzimage)
• -dtb: Specify the hardware description file (Device Tree Blob)
• --nographic: Run QEMU without GUI. It’s much more convenient.
• -append: Specify Linux kernel arguments. Here we set default console to ttyAMA0 which is one of QEMU’s console when Guest OS/Applications wants to print something on host’s terminal.
• -drive: Specify a drive for the image. It can be SD card, flash, etc. It’s the lowest level of drive API. We use if(interface) SD card with write back cache policy to save image access time.
• -sd: It is a higher level API to specify a drive. It’s equivalent to “-drive if=sd,file=”
• -net nic,macaddr=$macaddr: Specify the mac address
• -net tap,vlan=0,ifname=tap0: Use tap device for internet access
• -snapshot: Don’t write back to the original disk image.

The following example can run QEMU with internet access. Please refer to QEMU’s NAT and Arch’s QEMU networking for further details.

#!/bin/bash

#export an environment variable with format print
printf -v macaddr "52:54:%02x:%02x:%02x:%02x" $(( $RANDOM & 0xff)) $(( $RANDOM & 0xff )) $(( $RANDOM & 0xff)) $(( $RANDOM & 0xff ))

sudo qemu-system-arm \
    -M vexpress-a9 \
    -kernel ./zImage_arch \
    -dtb ./vexpress-v2p-ca9.dtb \
    --nographic \
    -append "root=/dev/mmcblk0 rw roottype=ext4 console=ttyAMA0" \
    -drive if=sd,driver=raw,cache=writeback,file=./arch_rootfs.ext4 \
    -net nic,macaddr=$macaddr \
    -net tap,vlan=0,ifname=tap0 \
    -snapshot

You can now execute QEMU with the following command to run your kernel:

~/$qemu-system-arm \
-M vexpress-a9 \
-dtb ./vexpress-v2p-ca9.dtb \
-kernel ./zImage_arch \
-append "root=/dev/mmcblk0 rw roottype=ext4 console=ttyAMA0" \
-drive if=sd,driver=raw,cache=writeback,file=./arch_rootfs.ext4 \
--nographic \
-snapshot

For Arch Linux user, to set up networking for QEMU, you need to set up bridge with netctl for your computer, then do follows:

# Switch to bridge. qemu_bridge is the netctl config file
~$netctl switch-to qemu_bridge
# Assign gateway IP for bridge (This enables host to ssh guest)
~$sudo ip addr add dev br0 192.168.0.254/24
# IPv4 forwarding (This enables guest to connect to global net. -o must be the name of  bridge you used)
~$sudo iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE
~$sudo sysctl -w net.ipv4.ip_forward=1

Create/edit the following two files for QEMU’s hook.

#!/bin/sh

echo "Executing /etc/qemu-ifup"
echo "Bringing up $1 for bridged mode..."
sudo /usr/bin/ip link set $1 up promisc on
echo "Adding $1 to br0..."
sudo /usr/bin/brctl addif br0 $1
sleep 2

 

#!/bin/sh

echo "Executing /etc/qemu-ifdown"
sudo /usr/bin/ip link set $1 down
sudo /usr/bin/brctl delif br0 $1
sudo /usr/bin/ip link delete dev $1

Then, make them executable by:

~$sudo chmod +x /etc/qemu-ifup /etc/qemu-ifdown

In Arch ARM, you can enable the network by setting proper IP and correct gateway of br0.

# Set IP
~$ip addr add 192.168.0.2/24 dev eth0
# Set gateway to br0(host)
~$ip route add default via 192.168.0.254 dev eth0
# Set DNS name server
~$echo "nameserver 8.8.8.8" >> /etc/resolv.conf
# Use IPv4 mode of ping
~$ping -4 8.8.8.8