Linux system boot time

Linux system boot time

Some applications have specific requirements for a system’s boot time. Often the system does not need to be immediately ready for all its tasks, but it should be ready for certain mission-critical tasks (e.g. accepting commands over Ethernet or displaying a user interface). This article provides a few methodologies and low-hanging fruit for improving boot time on Toradex System on Modules.

Note: A few tips mentioned in this article require recompiling the U-Boot, Kernel or rebuilding a root file system from scratch. Please refer to their respective articles on our developer website.

Before starting the optimization, we need an appropriate method to measure the boot time. If an exact end-to-end boot time is required, it might even be necessary to involve the hardware (e.g. GPIOs and an oscilloscope). In most cases simple monitoring of the serial port from a host system is accurate enough. A popular utility tool to monitor the timings of serial output is Tim Bird’s grabserial. This utility tool adds a timestamp to each line captured from the serial port as shown below:

The first number represents the time stamp (since the first character was received) while the second number shows the delta between the timestamps of the current and the last line.

This article is generally applicable to all of our modules. However, I do present some measurements and improvements specifically using our NXP ® /Freescale Vybrid-based module — Colibri VF61.

There are roughly three phases of a Linux system boot, which we are listed below and will be examined during the course of this blog.

Boot loader Linux kernel User space (init system)

Boot loader
There are actually two more phases before the boot loader can run: Hardware initialization and boot ROM. The hardware initialization phase is needed to fulfil power sequencing requirements and bus or SoC reset timing requirements. This phase is usually fixed and in the range of 10-200ms. Arm SoCs boot from a firmware stored on an internal ROM. This firmware loads the boot loader from the boot media. The runtime is usually rather short and influenced by the boot loader’s size. Other than minimizing the boot loader’s size, optimizations are rather hard. Real optimization potential and flexibility are within the boot loader (U-Boot).

With the current release V2.5 Beta 1, the time from the first character to the Kernel start is

1.85 seconds. This involves the following steps:

110ms, measured from the first character received) Autoboot delay (1s) UBI initialization and UBIFS mount (

300ms, thanks to a feature called Fastmap. Without Fastmap it would take around 1.6s) Loading the kernel (375ms) Loading and patching the device tree (

35ms) And finally jump into the kernels start address

The obvious optimization is reducing the Autoboot delay. This can be set to zero using:

This can also be configured as a default by using the CONFIG_BOOTDELAY config symbol. But in the current release, with a bootdelay of 0, there is no way to get into the boot loader’s console directly. U-Boot provides an option called CONFIG_ZERO_BOOTDELAY_CHECK which will check for one character even if the bootdelay is 0. We have added this option to our default configuration for the next release.

Serial output is sent synchronously. This means that the CPU waits until the character has been sent over the serial line. Therefore, each character that is printed slows down the U-Boot. Especially since UBI prints a lot of information messages, there is potential for optimization. It turns out that there is a config symbol CONFIG_UBI_SILENCE_MSG.

Ensuring that the hardware is used as efficiently as possible needs insight into what the hardware is capable of and what is currently being implemented. A missing feature till now was the Level 2 Cache (only on Colibri VF61). After implementing Level 2 cache, the boot time improved by more than 40ms.

Removing certain features helps to decrease the relocation time and initialization of such features. By removing Display support (DCU), EXT3 and EXT4 support as well as USB peripheral drivers such as DFU and mass storage. It helped us to decrease the size of U-Boot to 366kB and shaved away another 10ms.

According to the timestamps, most of the time is spent in attaching UBI and mounting the UBIFS as well as loading the kernel (

380ms). Obviously, the kernel size and the load time correlate linearly hence optimizing the kernel size will help to improve the boot time further.

Kernel
To measure the kernel boot time only, the “match” feature of grabserial can be used to reset the timestamp in the last message printed by the boot loader:

The end of the boot time is somewhat hard to determine, since the kernel continues to initialize hardware even after the root file system has been mounted and the first user space process (init) starts running (delayed initialization). The string “Freeing unused kernel memory” is the last message emitted before the init process is started, and hence marks the end of the kernels “linear” init procedure (see kernel_init in init/main.c). We will use the timestamp of that message to compare boot times. The shipped kernel has a zipped size of 4316kB and a boot time of 2.56 seconds.

Similar to U-Boot, the Linux kernel prints all messages synchronously to the serial console. The exact behaviour depends on the serial console used, but the LPUART (the driver for Vybrid’s console) waits synchronously until the character is sent over the serial port. This has the advantage that when the kernel crashes, all the messages up to that point are visible. If the messages were sent asynchronously, the last visible message would not indicate the location of a crash…

The kernel has an argument to minimize the amount of kernel messages displaying: “quiet”. However, this also silences our anchor for the boot time measurement (“Freeing unused kernel memory”). The easiest way to get the message back on the screen is to elevate the log level for that particular print statement. It is located in ‘mm/page_alloc.c’ — search for “Freeing %s memory”. I elevated the message to ‘pr_alert’. The measurement showed an improvement of 1.55 seconds, which is an improvement greater than factor of 2!

The easiest way to archive further improvements is by removing features. The Yocto project has a handy tool called ksize.py which needs to be started from within a kernel build directory. The tool prints tables identifying the size of individual kernel parts. The first table shows a high level overview (use make clean before building to get an accurate overview):

Which features can be removed safely is obviously application specific. Going through the individual high level directories helps to quickly remove the most promising candidates. For this article I removed several file systems (cifs, nfs, ext4, ntfs), the audio subsystem, multimedia support, USB and wireless network adapter support. The kernel ended up at about 3356kB, roughly 1MB less than before. This also decreased the kernel loading time in the boot loader by about

Another improvement idea can be to evaluate different compression algorithm, even though the current default algorithm in our kernel configuration is LZO which is already quite elaborate.

User Space
In Linux user space, initialization is done by the init system. The Toradex BSP images use the Ångström standard init system which is systemd. Systemd, the de facto standard init system on the Linux desktop nowadays, is very feature-rich and is especially designed with dynamic systems in mind. Systemd also addresses boot time. Multiple daemons are started simultaneously (leveraging today’s multi-core system,); socket activation allows delayed loading of services at a later point in time and device activation allows starting services on demand. Furthermore, the integrated logging daemon journald saves space due to binary-packed log files and sophisticated log file management.

Depending on the application, an embedded system might be rather static. Hence, the dynamic features of systemd are not really needed. Unfortunately systemd is not very modular, or the individual modules have interlocked dependencies. This makes it hard to strip down systemd to a bare minimum. This section is separated into two parts: the first part shows systemd boot optimization techniques; whereas, the second part looks at System V and other alternatives.

In both parts we use the “Freeing unused kernel memory” message as the base time for time measurement:

systemd
For this blog post, we define the login shell on the serial console as a critical task. The login shell is defined as “Type=Idle”, which means that by definition, it starts only after all services have been started.

To start a headless or framebuffer-based application, one would typically create a new service. Systemd allows defining certain requirements as service needs before it can be started (e.g. Network with “Wants=network-online.target”) and then automatically ensures that the services gets started as soon as the requirements are met. However, since services are started in parallel, the CPU resources get shared amongst them. But still, the application is likely up and running before the serial console comes available, hence the following numbers may appear to be be on the higher side.

The quiet argument in the kernel arguments is also picked up by systemd. This change already has a positive effect on the systemd boot time, shaving off about 1.6s in the process.

systemd provides an utility called systemd-analyze which prints a list of services and their starting time when initiated with the “blame” argument. This allows finding boot time offenders quite easily; however, the values might be misleading since the time is measured according to the wall clock time. A listed service might just be in the sleep state the CPU is processing other work. So the service at the top of the list may not be the biggest boot time offender, especially on single core system.

Services can be disabled using the disable commands. Some services (especially the services provided by systemd itself) might need the mask command to disable them. Some might still be required for the system to operate; hence disabling theservice should be done carefully and only one at a time. For this article, the following services have been disabled:

Systemd comes with its own system logging daemon called journald. It is one of those components that should not be disabled entirely. During booting up the logging daemon needs to manage and delete old log files on the disk as well as write new log entries to the disk. By disabling the logging in to the disk boot time can already be improved, with the cost of having no log files stored of course. Use Storage=none in /etc/systemd/journald.conf to disable the log storage part.

System V init and other alternatives
For many years SysV has been the standard init system also on Linux. Due to its script based init system, it is very modular and relatively easy to strip to a bare minimum. Especially for relatively static systems, where systemd’s device activation or socket activation are not needed, SysV is a good alternative.

The Yocto project’s reference distribution “poky”, which I blogged about in my last article[The Yocto Project’s Reference Distribution “Poky” on Toradex Hardware], uses SysV by default. Using the ‘minimal-console-image’ and a static IP address configuration, the measured user space boot time on Colibri VF61 is

Источник

System time

This article or section needs expansion.

In an operating system, the time (clock) is determined by three parts: time value, whether it is local time or UTC or something else, time zone, and Daylight Saving Time (DST) if applicable. This article explains what they are and how to read/set them. Two clocks are present on systems: a hardware clock and a system clock which are also detailed in this article.

Standard behavior of most operating systems is:

• Set the system clock from the hardware clock on boot.
• Keep accurate time of the system clock, see #Time synchronization.
• Set the hardware clock from the system clock on shutdown.

Contents

Hardware clock

The hardware clock (a.k.a. the Real Time Clock (RTC) or CMOS clock) stores the values of: Year, Month, Day, Hour, Minute, and Seconds. Only 2016, or later, UEFI firmware has the ability to store the timezone, and whether DST is used.

Read hardware clock

Set hardware clock from system clock

The following sets the hardware clock from the system clock. Additionally it updates /etc/adjtime or creates it if not present. See hwclock(8) section «The Adjtime File» for more information on this file as well as the #Time skew section.

System clock

The system clock (a.k.a. the software clock) keeps track of: time, time zone, and DST if applicable. It is calculated by the Linux kernel as the number of seconds since midnight January 1st 1970, UTC. The initial value of the system clock is calculated from the hardware clock, dependent on the contents of /etc/adjtime . After boot-up has completed, the system clock runs independently of the hardware clock. The Linux kernel keeps track of the system clock by counting timer interrupts.

Read clock

To check the current system clock time (presented both in local time and UTC) as well as the RTC (hardware clock):

Set system clock

To set the local time of the system clock directly:

sets the time to May 26th, year 2014, 11:13 and 54 seconds.

Time standard

There are two time standards: localtime and Coordinated Universal Time (UTC). The localtime standard is dependent on the current time zone, while UTC is the global time standard and is independent of time zone values. Though conceptually different, UTC is also known as GMT (Greenwich Mean Time).

The standard used by the hardware clock (CMOS clock, the BIOS time) is set by the operating system. By default, Windows uses localtime, macOS uses UTC, other UNIX and UNIX-like systems vary. An OS that uses the UTC standard will generally consider the hardware clock as UTC and make an adjustment to it to set the OS time at boot according to the time zone.

If multiple operating systems are installed on a machine, they will all derive the current time from the same hardware clock: it is recommended to adopt a unique standard for the hardware clock to avoid conflicts across systems and set it to UTC. Otherwise, if the hardware clock is set to localtime, more than one operating system may adjust it after a DST change for example, thus resulting in an over-correction; problems may also arise when traveling between different time zones and using one of the operating systems to reset the system/hardware clock.

The hardware clock can be queried and set with the timedatectl command. You can see the current hardware clock time standard of the Arch system using:

To change the hardware clock time standard to localtime, use:

To revert to the hardware clock being in UTC, type:

These generate /etc/adjtime automatically and update the RTC accordingly; no further configuration is required.

During kernel startup, at the point when the RTC driver is loaded, the system clock may be set from the hardware clock. Whether this occurs depends on the hardware platform, the version of the kernel and kernel build options. If this does occur, at this point in the boot sequence, the hardware clock time is assumed to be UTC and the value of /sys/class/rtc/rtcN/hctosys (N=0,1,2. ) will be set to 1.

Later, the system clock is set again from the hardware clock by systemd, dependent on values in /etc/adjtime . Hence, having the hardware clock using localtime may cause some unexpected behavior during the boot sequence; e.g system time going backwards, which is always a bad idea (there is a lot more to it). To avoid it systemd will only synchronize back, if the hardware clock is set to UTC and keep the kernel uninformed about the local timezone. As a consequence timestamps on a FAT filesystem touched by the Linux system will be in UTC.

UTC in Microsoft Windows

To dual boot with Windows it is recommended to configure Windows to use UTC, rather than Linux to use localtime. (Windows by default uses localtime [1].)

It can be done by a simple registry fix: Open regedit and add a DWORD value with hexadecimal value 1 to the registry:

You can do this from an Administrator Command Prompt running:

Alternatively, create a *.reg file (on the desktop) with the following content and double-click it to import it into registry:

Should Windows ask to update the clock due to DST changes, let it. It will leave the clock in UTC as expected, only correcting the displayed time.

The #Hardware clock and #System clock time may need to be updated after setting this value.

If you are having issues with the offset of the time, try reinstalling tzdata and then setting your time zone again:

Historical notes

For really old Windows, the above method fails, due to Windows bugs. More precisely,

• For 64-bit versions of Windows 7 and older builds of Windows 10, there was a bug that made it necessary to have a QWORD value with hexadecimal value of 1 instead of a DWORD value. This bug has been fixed in newer builds and now only DWORD works.
• Before Vista SP2, there is a bug that resets the clock to localtime after resuming from the suspend/hibernation state.
• For XP and older, there is a bug related to the daylight saving time. See [2] for details.
• For even older versions of Windows, you might want to read https://www.cl.cam.ac.uk/

mgk25/mswish/ut-rtc.html — the functionality was not even documented nor officially supported then.

For these operating systems, it is recommended to use localtime.

UTC in Ubuntu

Ubuntu and its derivatives have the hardware clock set to be interpreted as in «localtime» if Windows was detected on any disk during Ubuntu installation. This is apparently done deliberately to allow new Linux users to try out Ubuntu on their Windows computers without editing the registry.

For changing this behavior, see above.

Time zone

To check the current zone defined for the system:

To list available zones:

To set your time zone:

This will create an /etc/localtime symlink that points to a zoneinfo file under /usr/share/zoneinfo/ . In case you choose to create the link manually (for example during chroot where timedatectl will not work), keep in mind that it must be a symbolic link, as specified in archlinux(7) [dead link 2021-02-08] :

Setting based on geolocation

To set the timezone automatically based on the IP address location, one can use a geolocation API to retrieve the timezone, for example curl https://ipapi.co/timezone , and pass the output to timedatectl set-timezone for automatic setting. Some geo-IP APIs that provide free or partly free services are listed below:

Update timezone every time NetworkManager connects to a network

Alternatively, the tool tzupdate AUR automatically sets the timezone based on the geolocation of the IP address. This comparison of the most popular IP geolocation apis may be helpful in deciding which API to use in production.

Time skew

Every clock has a value that differs from real time (the best representation of which being International Atomic Time); no clock is perfect. A quartz-based electronic clock keeps imperfect time, but maintains a consistent inaccuracy. This base ‘inaccuracy’ is known as ‘time skew’ or ‘time drift’.

When the hardware clock is set with hwclock , a new drift value is calculated in seconds per day. The drift value is calculated by using the difference between the new value set and the hardware clock value just before the set, taking into account the value of the previous drift value and the last time the hardware clock was set. The new drift value and the time when the clock was set is written to the file /etc/adjtime overwriting the previous values. The hardware clock can therefore be adjusted for drift when the command hwclock —adjust is run; this also occurs on shutdown but only if the hwclock daemon is enabled, hence for Arch systems which use systemd, this does not happen.

If the hardware clock keeps losing or gaining time in large increments, it is possible that an invalid drift has been recorded (but only applicable, if the hwclock daemon is running). This can happen if you have set the hardware clock time incorrectly or your time standard is not synchronized with a Windows or macOS install. The drift value can be removed by first removing the file /etc/adjtime , then setting the correct hardware clock and system clock time. You should then check if your time standard is correct.

The software clock is very accurate but like most clocks is not perfectly accurate and will drift as well. Though rarely, the system clock can lose accuracy if the kernel skips interrupts. There are some tools to improve software clock accuracy:

Time synchronization

The Network Time Protocol (NTP) is a protocol for synchronizing the clocks of computer systems over packet-switched, variable-latency data networks. The following are implementations of NTP available for Arch Linux:

• Chrony — A client and server that is roaming friendly and designed specifically for systems that are not online all the time.

https://chrony.tuxfamily.org/ || chrony

• ConnMan — A lightweight network manager with NTP support.

https://01.org/connman (waybackmachine) || connman

• Network Time Protocol daemon — The reference implementation of the protocol, especially recommended to be used on time servers. It can also adjust the interrupt frequency and the number of ticks per second to decrease system clock drift, and will cause the hardware clock to be re-synchronised every 11 minutes.

https://www.ntp.org/ || ntp

• ntpclient — A simple command-line NTP client.

http://doolittle.icarus.com/ntpclient/ || ntpclientAUR

• NTPsec — A fork of NTPd, focused on security.

https://ntpsec.org/ || ntpsecAUR

• OpenNTPD — Part of the OpenBSD project and implements both a client and a server.

https://www.openntpd.org/ || openntpd

• sntp — An SNTP client that comes with NTPd. It supersedes ntpdate and is recommended in non-server environments.

https://www.ntp.org/ || ntp

• systemd-timesyncd — A simple SNTP daemon that only implements a client side, focusing only on querying time from one remote server. It should be more than appropriate for most installations.

https://www.freedesktop.org/wiki/Software/systemd/ || systemd

Per-user/session or temporary settings

For some use cases it may be useful to change the time settings without touching the global system values. For example to test applications relying on the time during development or adjusting the system time zone when logging into a server remotely from another zone.

To make an application «see» a different date/time than the system one, you can use the faketime(1) utility (from libfaketime ).

If instead you want an application to «see» a different time zone than the system one, set the TZ environment variable, for example:

This is different than just setting the time, as for example it allows to test the behavior of a program with positive or negative UTC offset values, or the effects of DST changes when developing on systems in a non-DST time zone.

Another use case is having different time zones set for different users of the same system: this can be accomplished by setting the TZ variable in the shell’s configuration file, see Environment variables#Defining variables.

Troubleshooting

Clock shows a value that is neither UTC nor local time

This might be caused by a number of reasons. For example, if your hardware clock is running on local time, but timedatectl is set to assume it is in UTC, the result would be that your timezone’s offset to UTC effectively gets applied twice, resulting in wrong values for your local time and UTC.

To force your clock to the correct time, and to also write the correct UTC to your hardware clock, follow these steps:

• Setup ntpd (enabling it as a service is not necessary).
• Set your time zone correctly.
• Run ntpd -qg to manually synchronize your clock with the network, ignoring large deviations between local UTC and network UTC.
• Run hwclock —systohc to write the current software UTC time to the hardware clock.

Tips and tricks

fake-hwclock

alarm-fake-hwclock designed especially for system without battery backed up RTC, it includes a systemd service which on shutdown saves the current time and on startup restores the saved time, thus avoiding strange time travel errors.

Источник