Qemu run linux kernel

Nick Desaulniers

The enemy’s gate is down

Booting a Custom Linux Kernel in QEMU and Debugging It With GDB

Oct 24 th , 2018 | Comments

Typically, when we modify a program, we’d like to run it to verify our changes. Before booting a compiled Linux kernel image on actual hardware, it can save us time and potential headache to do a quick boot in a virtual machine like QEMU as a sanity check. If your kernel boots in QEMU, it’s not a guarantee it will boot on metal, but it is a quick assurance that the kernel image is not completely busted. Since I finally got this working, I figured I’d post the built up command line arguments (and error messages observed) for future travelers. Also, QEMU has more flags than virtually any other binary I’ve ever seen (other than a google3 binary; shots fired), and simply getting it to print to the terminal is ¾ the battle. If I don’t write it down now, or lose my shell history, I’ll probably forget how to do this.

Booting in QEMU

We’ll play stupid and see what errors we hit, and how to fix them. First, let’s try just our new kernel:

A new window should open, and we should observe some dmesg output, a panic, and your fans might spin up. I find this window relatively hard to see, so let’s get the output (and input) to a terminal:

This is missing an important flag, but it’s important to see what happens when we forget it. It will seem that there’s no output, and QEMU isn’t responding to ctrl+c . And my fans are spinning again. Try ctrl+a , then c , to get a (qemu) prompt. A simple q will exit.

Next, We’re going to pass a kernel command line argument. The kernel accepts command line args just like userspace binaries, though usually the bootloader sets these up.

Well at least we’re no longer in the dark (remember, ctrl+a , c , q to exit). Now we’re panicking because there’s no root filesystem, so there’s no init binary to run. Now we could create a custom filesystem image with the bare bones (definitely a post for another day), but creating a ramdisk is the most straightforward way, IMO. Ok, let’s create the ramdisk, then add it to QEMU’s parameters:

Unfortunately, we’ll (likely) hit the same panic and the panic doesn’t provide enough information, but the default maximum memory QEMU will use is too limited. -m 512 will give our virtual machine enough memory to boot and get a busybox based shell prompt:

Enabling kvm seems to help with those fans spinning up:

Finally, we might be seeing a warning when we start QEMU:

Just need to add -cpu host to our invocation of QEMU. It can be helpful when debugging to disable KASLR via nokaslr in the appended kernel command line parameters, or via CONFIG_RANDOMIZE_BASE not being set in our kernel configs.

We can add -s to start a gdbserver on port 1234, and -S to pause the kernel until we continue in gdb.

Attaching GDB to QEMU

Now that we can boot this kernel image in QEMU, let’s attach gdb to it.

If you see this on your first run:

Then you can do this one time fix in order to load the gdb scripts each run:

Now that QEMU is listening on port 1234 (via -s ), let’s connect to it, and set a break point early on in the kernel’s initialization. Note the the use of hbreak (I lost a lot of time just using b start_kernel , only for the kernel to continue booting past that function).

We can start/resume the kernel with c , and pause it with ctrl+c . The gdb scripts provided by the kernel via CONFIG_GDB_SCRIPTS can be viewed with apropos lx . lx-dmesg is incredibly handy for viewing the kernel dmesg buffer, particularly in the case of a kernel panic before the serial driver has been brought up (in which case there’s output from QEMU to stdout, which is just as much fun as debugging graphics code (ie. black screen)).

Where to Go From Here

Maybe try cross compiling a kernel (you’ll need a cross compiler/assembler/linker/debugger and likely a different console=ttyXXX kernel command line parameter), building your own root filesystem with buildroot, or exploring the rest of QEMU’s command line options.

Источник

chrisdone / gist:02e165a0004be33734ac2334f215380e

Configure the kernel according the following:

Build the kernel:

Configure Busybox according the following:

Create an initramfs:

Add a $INITRAMFS_BUILD/init script to the initramfs with the following content:

Create the initramfs archive:

Run and see ( a x to quit)

Note: for better performance, add the -enable-kvm option if your host has KVM enabled:

We assume that a toolchain is available in /opt/toolchains/x86_64-unknown-linux-gnu with prefix x86_64-unknown-linux-gnu , gcc version 5.x, kernel headers series 4.3.x, glibc C library and C++ support. These are reasonable defaults if you are using a toolchain generated by crosstool-NG. Adapt to your own situation. Notes:

• You cannot use the native toolchain of your host computer (see Buildroot documentation to understand why).
• If you do not have a toolchain already, you can build one using crosstool-NG (or Buildroot itself) and reuse it for other projects.
• crosstool-NG is the recommended tool to build your own toolchain but avoid using uClibc (no IPV6 support), prefer uClibc-ng or glibc.
• You can also use the built-in toolchain of Buildroot but be aware that it will take way longer than using an existing toolchain. Be also aware that in many cases you will have to re-build the toolchain after modifying the Buildroot configuration.
• No yet convinced? Please use crosstool-NG, build and use your own toolchain.

Configure Buildroot according the following:

Save the configuration and build:

Add a $BUILDROOT_BUILD/overlay/init script to the overlay with the following content:

Build the root filesystem:

Run and see ( a x to quit)

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

Add and run a custom user application

Create a new directory for the custom user applications:

Add an application source file $APPS/hello_world.c with the following content:

Add a $APPS/Makefile with the following content (replace the CROSS_COMPILE definition with whatever is appropriate in your specific case):

Compile the application, copy it in the Buildroot overlay directory and re-build the root filesystem:

Run and see ( a x to quit)

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

Add loadable module support to the Linux kernel

Change the kernel configuration according the following:

Re-build the kernel and its modules (none, in our case) and install the modules in the Buildroot overlay directory:

Add a custom user module

Create a new directory for the custom user modules:

Add a module source file $MODULES/hello_world.c with the following content:

Add a $MODULES/Makefile with the following content:

Compile the module, install it in the Buildroot overlay directory and re-build the root filesystem:

Run and see ( a x to quit)

Note: for better performance, add the -enable-kvm option if your host has KVM enabled.

Источник

Prepare the environment for developing Linux kernel with qemu.

Oct 4, 2019 · 10 min read

I would share the environment for developing linux kernel with Qemu. I think the Qemu is powerful hardware emulator to run Linux kernel for developer. This ables you to run Linux kernel without real hardware and also provides features to analyze Linux kernel itself.

OS what I used for preparing the environment to develop Linux kernel is Ubuntu 18.04 and I will install qemu for x86_64 guest. It has advantage of performance with KVM if the architecture is same with host and qemu guest.

Start following order:

1. Install packages releated to building Linux kernel and running it on Qemu.
2. Clone Linux kernel source code, configure and build for running on qemu-x86_64 guest.
3. GDB to set break point
4. buildroot is for making root file system.

Install packages

Install git pack a ges to clone the Linux Kernel source code from the linux-next branch in mainline of Linux Kernel Repository

Install packages to build linux kernel from the source code.

Clone Linux Kernel Source from the development branch.

Linux kernel has ‘linux-next’ branch to take patches from developers. You can take a look at the tree with this link. If you want to fix issues or adds features to the Linux kernel, you should get this(linux-next) branch and remain latest. There are lots of subsystem maintainer’s branches but linux-next branch is good to start Linux Kernel development.

If you are tracking linux-next branch with git, you should not use git pull. Instead, you should use git fetch origin and git reset —hard remotes/origin/master.

These set latest released source code in master (local) branch.

Configure the Linux Kernel

Configure the Linux kernel to build kernel image that is able to run on qemu(x86_64 architecture). You can find config file what is used as base for x86_64 architecture in arch/x86/configs/x86_64_defconfig. Let’s find out how to .config file as default configuration file for x86_64 architecture.

You can check the .config file is generated and it will be used in kernel compile. But there are some options what are added to .config for GDB debugger.

Above command will open TUI(Text-based UI) to help you configure the linux kernel easly and also load current .config file automatically.

Some options should be enabled to debug kernel with qemu. Move cursor to “Kernel hacking” and enter into it.

Select “Compile the kernel with debug info” option first, and then show more options related to kernel debug. Need to select “Provide GDB scripts for kernel debugging (New)” option. But you should leave “Reduce debugging information” option off.

Exit kernel configuration with selecting “ ” menu in bottom of menu. And save it with your selection.

Now we can compile the Linux kernel with “make” command.

If the build is done, you can see the message like below:

Install the qemu and run it with kernel image

You are ready to run Linux kernel on qemu, but before, you should install packages to run qemu.

Then, you can use the “qemu-system-x86_64″ tool for running our kernel image what compiled in previous step.

I just use pseudo console to show the kernel logs instead of using virtual display. If it works properly, you can see the kernel messages on your terminal.

Above execution has no file system image which could be mounted in qemu virtual environment, you can see the kernel panic message related to mounting file system. no wrorry about that. And “Ctrl + c” key will exit from the qemu.

Access running qemu system and debug the kernel with GDB

You need to “gdb” tool for accessing running linux kernel on Qemu. Install it following command:

There are two options what you should add it to qemu before running. One is “-s”(lowercase ) that option means use the port 1234 as tcp(tcp::1234) to connect it through gdb. Another is “-S” (uppercase) that option means stop cpu until continue from gdb what is connected to tcp 1234 port.

There is one more point to take care before debugging with gdb. You should turn off the KASLR option by adding “nokaslr” to the kernel command line. Take a look after “-append”, you can see the “nokaslr”. If you want to know more details about KASLR, visit this article.

If you run above command, qemu will wait to be continued by gdb. Qemu will give some warning message but you don’t need to care about it.

Try to open another terminal to run gdb.

Type “target remote localhost:1234” in gdb shell.

You can connect qemu through localhost:1234 and then you can break start point of kernel(start_kernel function is first C language code) with ‘break’ command of gdb.

Start kernel by putting ‘continue’ command of gdb, and break the kernel right after. The terminal that qemu was excuted didn’t any changes because the ‘start_kernel’ function is early stage of linux kernel so there is nothing to show kernel message. You can check the source code in gdb terminal.

The kernel is stopped in main.c:575. You can use ’n’ (next), ‘c’ (continue) and so on. Put another ‘continue’ command, then you can see the messages in qemu executed terminal. If you didn’t set other break point to debug, the gdb could not show ‘(gdb)’ and wait qemu to be exited.

Create Root file system from buildroot project

Buildroot helps you making root file system easly and created root file system image will be mounted in qemu system. The buildroot is normally used for embedded system but I think it is good enough to debug the linux kernel with qemu.

First, clone the buildroot project.

Second, configure buildroot option with ‘make menuconfig’.

Select “Target Options” to run it on x86_64 virtualized system with qemu.

Target Options -> Target Architecture →

Filesystem images → ext2/3/4 root file system

Choose ext4 file system to be mounted as default file system for the system.

Build ‘buildroot’ project with ‘make’

The rootfs.ext4 has been created, so you can use the generated image could be used as root file system in qemu. Check the output file in buildroot/output/images/ directory. This image file will be used in qemu execution.

Run qemu with own Linux Kernel and Root file system

Test rootfs.ext4 image file that it works properly with qemu. Run qemu (above) with rootfs.ext4.(Please check the path of rootfs.ext4 file) Then you will see the login shell in last kernel message.

Buildroot has ‘root’ as default login user without password. Just put root after buildroot login: then you will get the shell.

Now, you can connect qemu with gdb. It means you can trace the source code any kernel features like file system, scheduler and memory management. It helps you study the inside of Linux kernel. Try set break point of source code what you want to analyze flow of some function. Note that you should use ‘-s’ and ‘-S’ options in qemu to debugging Linux kernel.

2. Run gdb and connect qemu in gdb shell

When you put ‘target remote :1234’, the qemu stops in somewhere of linux kernel and print breaked point. And set another break point. I tried to set break point in mm/page_alloc.c:4767 (in __get_free_pages() function) and continue kernel with c command.

After put ‘c’ command in gdb, you can input ‘root’ as user to login buildroot shell.

And nothing happended in qemu, but you can see the message in gdb shell. You can input command gdb shell. Just put list command. You will get the source code what you set break point before.

Conclusion

I think the Linux kernel is really hard to start studying. Because I could not figure out where to start the code looking and also I want to see the values and status when some function is called like __get_free_pages(). So if you use qemu and gdb, these make you easy to read the source code and easy to check the values while the function is handled.

Источник