How to run aarch64 binary on host PC

Linux

This article explains how binaries of the Jetson Nano platform, an aarch64 architecture, are executed on a host PC with an x86_64 architecture.

binfmt_misc

binfmt_misc is a Linux mechanism that allows you to execute arbitrary files directly. By “directly” is meant that if you specify ./test.sh from the shell to execute test.sh. On the other hand, bash test.sh executes test.sh via bash, so it is not direct.

$ echo -e '#!/bin/shn/usr/bin/echo hoge' > test.sh
$ chmod +x test.sh
$ ./test.sh
hogeCode language: Bash (bash)

The figure below shows the sequence of execution of the above test.sh by shell.

  1. shell(PID=10) calls fork or clone to create a child process shell(PID=11)
  2. shell(PID=11) calls execve(“. /test.sh”)
  3. The kernel use the handler for test.sh if it is found, and then load the binaries.
  4. test.sh(PID=11) calls fork or clone to create a child process test.sh(PID=12)
  5. test.sh(PID=12) calls execve(“/usr/bin/echo”)
  6. The kernel use the handler for ./usr/bin/echo if it is found, and then load the binaries

The kernel usually has bifmt_script to load scripts, binfmt_elf to load ELF binaries (executable, library files) etc, and if the first two bytes of the executable specified in execve are “#!“, then binfmt_script loads the interpreter following the two byte of “!#“. If it is determined that the first few bytes indicate and ELF binary, binfmt_elf loads ELF file.

This is where binfmt_misc finally comes in. binfmt_misc, like binfmt_script, works by executing a specific interpreter if a specific byte at the beginning matches.

Install binfmt_misc.

sudo apt update && sudo apt install -y binfmt-supportCode language: Bash (bash)

As a sample, we will create a binfmt_misc that will start the image viewer /usr/bin/eog when a JPEG is executed directly.

$ ls /proc/sys/fs/binfmt_misc
python3.8  register  status
$ sudo bash -c  "echo ':jpeg:M::xffxd8::/usr/bin/eog:' > /proc/sys/fs/binfmt_misc/register"
# make sure "jpeg" is created
$ ls /proc/sys/fs/binfmt_misc
jpeg  python3.8  register  status
$ cat /proc/sys/fs/binfmt_misc/jpeg
enabled
interpreter /usr/bin/eog
flags: 
offset 0
magic ffd8Code language: Bash (bash)

First of all, /proc/sys/fs/infmt_misc is not a normal file system, it is a pseudo file system created by the kernel and its file size is always zero.

Writing data according to the following format to /proc/sys/fs/binfmt_misc/register will register an entry for binfmt_misc in the kernel, and the registered entry will be created under /proc/sys/fs/binfmt_misc.

:name:type:offset:magic:mask:interpreter:flagsCode language: plaintext (plaintext)

name is the name of the entry to be registered, type is the file extension if E is specified, or byte sequence if M is specified, and interpreter is the program to be executed. For more details, see https://www.kernel.org/doc/Documentation/admin-guide/binfmt-misc.rst. In the above JPEG example, name is jpeg, type is M, magic is ffd8, and interpreter is /usr/bin/eog. After writing to register, we will see a file named jpeg is created under /proc/sys/fs/binfmt_misc.

Let’s execute the JPEG directly.

wget http://www.ess.ic.kanagawa-it.ac.jp/std_img/colorimage/Lenna.jpg
chmod +x Lenna.jpg
./Lenna.jpgCode language: Bash (bash)

qemu-user-static

Now that we understand how binfmt_misc works, we will install qemu-user-static.

$ sudo apt install qemu-user-static
$ ls /proc/sys/fs/binfmt_misc/
jpeg          qemu-arm    qemu-m68k        qemu-mips64el   qemu-ppc         qemu-riscv32  qemu-sh4eb        qemu-xtensa
python3.8     qemu-armeb  qemu-microblaze  qemu-mipsel     qemu-ppc64       qemu-riscv64  qemu-sparc        qemu-xtensaeb
qemu-aarch64  qemu-cris   qemu-mips        qemu-mipsn32    qemu-ppc64abi32  qemu-s390x    qemu-sparc32plus  register
qemu-alpha    qemu-hppa   qemu-mips64      qemu-mipsn32el  qemu-ppc64le     qemu-sh4      qemu-sparc64      status
Code language: Bash (bash)

We can see that some files have been added under /proc/sys/fs/binfmt_misc. Let’s have a look at a file called qemu-aarch64.

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffffCode language: Bash (bash)

This means that “if the first 20 bytes of the file (the result of bitwise logical multiplication of the file and the mask) match magic, then execute the file with /usr/bin/qemu-aarch64-static as interpreter. magic’s value is aarch64 ELF binary.

/usr/bin/qemu-aarch64-static is an emulator that converts aarch64 instructions to x86_64 instructions and executes them. For more information, please refer to https://www.qemu.org/docs/master/user/main.html. It also describes the conversion of system call numbers between aarch64 Linux and x86_64 Linux, and the handling of atomic instructions that cannot be emulated.

First, create a program on the Jetson Nano.

$ cat hello.c
#include <stdio.h>

int main(int argc, char *argv[]) {
	printf("hello worldn");
	return 0;
}
$ gcc hello.c -o hello --staticCode language: C++ (cpp)

Build with --static. Copy the created hello binary to the host PC and execute the following.

$ file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=cc6b82b4c3ee848187cbdc1483af59440970480f, not strippedCode language: Bash (bash)

We can confirm that it is built with aarch64. Let’s execute it.

$ ./hello
hello worldCode language: Bash (bash)

/usr/bin/qemu-aarch64-static was invoked as an interpreter by binfmt_misc and hello was executed.

That’s all

Reference