What is this about?

Operating systems can be a very complex topic. Learning how operating systems work can be a great learning experience.

The purpose of this series is to teach the black art of Operating System (OS) Development, from the ground up. Whether you want to make your own OS, or simply to learn how they work, this series is for you.

What is an Operating System?

An Operating System provides the basic functionality, look, and feel, for a computer. The primary purpose is to create a workable Operating Environment for the user.

An example of an Operating System is Windows, Linux, and Macintosh.

If you have never programmed before

Computer programming is designing and writing software, or programs, for the computer to load and execute. However, the Operating System needs to be designed with this functionality.

An Operating System is not a single program, but a collection of software that work and communicate with each other. This is what I mean by «Operating Environment».

Because Operating Systems are a collection of software, in order to develop an Operating System, one must know how to develop software. That is, one must know computer programming.

If you have never programmed before, take a look at the Requirements section below, and look no further. This section will have links to good tutorials and articles that could help you to learn computer programming with C++ and 80×86 Assembly Language.

Requirements

Knowledge of the C Programming Language

Using a high level language, such as C, can make OS development much easier. The most common languages that are used in OS development are C, C++, and Perl. Do not think these are the only languages that may be used; It is possible in other languages. I have even seen one with FreeBASIC! Getting higher level languages to work properly can also make it harder to work within the long run, however.

C and C++ are the most common, with C being the most used. C, as being a middle level language, provides high level constructs while still providing low level details that are closer to assembly language, and hence, the system. Because of this, using C is fairly easy in OS development. This is one of the primary reasons why it is the most commonly used: Because the C programming language was originally designed for system level and embedded software development.

Because of this, we are going to be using C for most of the OS.

C is a complex programming language, that can take a book to cover. If you do not know C, the following may help:

• Learn C – A complete resource for a beginner
• cprogramming.com
• Thinking in C++

I personally learned from the original «The C++ Programming language», which is now obsolete, though.

Knowledge of x86 Assembly Language

80×86 Assembly Language is a low level programming language. Assembly Language provides a direct one to one relation with the processor machine instructions, which make assembly language suitable for hardware programming.

Assembly Language, as being low level, tend to be more complex and harder to develop in, then high level languages like C. Because of this, and to aid in simplicity, We are only going to use assembly language when required, and no more.

Assembly Language is another complex language that can take a book to fill. If you do not know x86 Assembly Language, the following may help:

• Assembly Language: Step by Step
• Art of Assembly

I personally learned from Assembly Language Step by Step (Excellent beginning book) and the Art of Assembly Language. Both are very great books.

Getting ready

Tools of the trade

In developing low level code, we will need specialized low level software to help us out. Some of these tools are not needed, however, they are highly recommended as they can significantly aid in development.

NASM — The Assembler

The Netwide Assembler (NASM) can generate flat binary 16bit programs, while most other assemblers (Turbo Assembler (TASM), Microsoft’s Macro Assembler (MASM)) cannot.

During the development of the OS, some programs must be pure binary executables. Because of this, NASM is a great choice to use.

You can download NASM from here.

Microsoft Visual C++ 2005 or 2008

During OS Development, there are some things that we must have control over that not all compilers may support, however. For example, say good bye to all runtime compiler support (templates, exceptions) and the good old standard library! Depending on the design of your system, you may also need to support or change more detailed properties: Such as loading at a specific address, adding your own internal sections in your programs’ binary, etc..) The basic idea is that not all compilers out there are capable of developing operating system code.

I will be using Microsoft Visual C++ for developing the system. However, it is also possible to develop in other compilers such as DJGPP, GCC or even Cygwin. Cygwin is a command shell program that is designed to emulate Linux command shell. There is a GCC port for Cygwin.

You can get Visual C++ 2008 from here

You can also still get Visual C++ 2005 from here.

Support for other compilers

As previously stated, it is possible to develop an operating system using other compilers. While my primary compiler of use will be Visual C++, I will explain on how to setup the working environments so that you will be able to use your favorite compiler.

Currently, I plan on describing on setting up the environments for:

• DJGPP
• Microsoft Visual Studio 2005
• GCC

I will also try to support the following compilers, if possible:

Mingw

Pelles C

If you would like to add more to this list, please contact me.

Copying the Boot Loader

We can use NASM to assemble the program, but how do we get it on a floppy disk? We cannot just copy the file. Instead, we have to overwrite the boot record that Windows places (after formatting the disk) with our bootloader. Why do we need to do this? Remember that the BIOS only looks at the bootsector when finding a bootable disk. The bootsector, and the «boot record» are both in the same sector! Hence, we have to overwrite it.

There are alot of ways we can do this. Here, I will present two. If you are unable to get one method working on your system, our readers may try the other method.

Warning: Do Not attempt to play with the following software until I explain how to use it. Using this oftware incorrectly can corrupt the data on your disk or make your PC unable to boot.

PartCopy — Low Level Disk Copier

PartCopy allows the copying of sectors from one drive to another. PartCopy stands for «Partial copy». Its function is to copy a certain number of sectors from one location to another, to and from a specific address.

You can download it from here.

Windows DEBUG Command

Go to the command prompt, and type debug. You will be greeted by a little prompt (-):

You can have debug load a file into memory such as, say, our boot loader:

To write the file to the first sector of our disk, we need to use the W (Write) command which takes the following form:

Putting this together, this is our command to write boot_loader.bin to the boot sector of a floppy:

If you would like to learn more about this command, take a look at this tutorial.

VFD — Virtual Floppy Drive

Weather you have a floppy drive or not, this program is very useful. It can simulate a real floppy drive from a stored floppy image, or even in RAM. This program creates a virtual floppy image, allows formatting, and copying files (Such as, your kernel perhaps?) directly using Windows Explorer.

You can download it from here.

Bochs Emulator — PC Emulator and Debugger

You pop in a floppy disk into a computer, in hopes that it works. You boot your computer and look in aw at your greatest creation! . Until your floppy motor dies out because you forgot to send the command to the controller in your bootloader.

When working with low level code, it is possible to destroy hardware if you are not careful. Also, to test your OS, you will need to reboot your computers hundreds of times during development.

Also, what do you do if the computer just reboots? What do you do if your Kernel crashes? Because there is no debugger for your OS, it is virtually impossible to debug.

The solution? A PC Emulator. There are plenty available, two of them being VMWare and Bochs Emulator. I will be using Bochs and Microsoft Virtual PC for testing.

You can download Bochs from here.

Thats all, fokes

You do not need to know how to use the software I listed. I will explain how to use them as we start using them.

If you would like to run your system on a real computer that does not have a floppy drive, it is still possible to boot from CD even though it is a floppy image. This is done through Floppy Emulation that which most of BIOSs support.

Simply get a CD burning software (I personally use MagicISO) that can create a bootable ISO from a floppy image. Then, simply burn the ISO image to a CD and it should work.

The Build Process

There are a lot of tools listed above. To better understand how they can be useful, we should take a look at the entire build process of the OS.

• Setting everything up
  1. Use VFD to create and format a virtual floppy image to use.
  2. Set up Bochs Emulator to boot from the floppy image.
• The bootloader
  1. Assemble the bootloader with NASM to create a flat binary program.
  2. Use PartCopy or the DEBUG command to copy the bootloader to the bootsector of the virtual floppy image.
• The Kernel (And basically all other programs)
  1. Assembly and/or compile all sources into an object format (Such as ELF or PE) that can be loaded and executed by the boot loader.
  2. Copy kernel into floppy disk using Windows Explorer.
• Test it!
  1. Using Bochs emulator and debugger, using a real floppy disk, or by using MagicISO to create a bootable CD.

Until next time

Some of the terms and concepts listed here may be new to you. Do not worry—everything will be explained in the next few articles.

The purpose of this tutorial is to create a stepping stone for the rest of the series. It provides a basic introduction, and a listing of the tools we will be using. I will explain how to use these programs as we need to, so you do not need a tutorial on anything listed here besides what has been listed in the Requirements section.

We also have taken a look at the building process for developing an operating system. For the most part, its fairly simple, however it provides a way to see when the programs listed will be used.

In the next tutorial we are going to go back in time from the first Disk Operating System (DOS) and take a little tour through history. We will also look at some basic OS concepts.

We will not be using any of the tools listed above just yet, so you do not need to download them just yet.

Until next time,

Mike
BrokenThorn Entertainment. Currently developing DoE and the Neptune Operating System

Questions or comments? Feel free to Contact me.

Would you like to contribute and help improve the articles? If so, please let me know!

Getting Started

First of all, developing an operating system is probably one of the most challenging things you can do on a computer (next to killing the final boss in Doom on Nightmare difficulty level). Composing an operating system requires a lot of knowledge about several complex areas within computer science. You need to understand how hardware works and be able to read and write the complex assembly language as well as a higher-level language (such as C, C++, or Pascal). Your mind has to be able to wrap itself around abstract theory and hold a myriad of thoughts. Feel discouraged yet? Don’t fear! Because all of these things are also the things that makes OS programming fun and entertaining.

There is nothing like the feeling of accomplishment when you, finally, after hours of struggling, solve the problem. And after some time you are able to look back and see all of the things you’ve created from scratch. Your handwritten system is able to boot, performs magic against the hardware, and gives the user a user interface and programs to play with.

There is no absolute path you have to take when creating an OS. Once you get your initial system up and running (and you do this by finding appropriate tutorials), you choose the path you want to take next. Your OS is exactly that—yours. You have ultimate control, and the sky’s the limit!

Contents

The Hard Truth

Hopefully the basic fact that operating system development is a complicated and ongoing process does not discourage you. The truth is, operating system development is truly unparalleled since it requires the utmost amount of patience and careful code design, and it returns very little to no «instant gratification» you get from the development of things like games and web-based scripting.

You have been fairly warned of the hard work ahead, but if you are still interested then proceed forward into the realm of the operating system programmer. Prepare yourself for occasional bouts of confusion, discouragement, and for some of us. temporary insanity. In time, and with enough dedication, you will find yourself among the elite few who have contributed to a working operating system. If you do get discouraged along the way, refresh yourself with the content of this book. Hopefully it will remind you why you started such an insane journey in the first place.

At this stage, it would also pay to read the Beginner Mistakes page. Users on the forum have noticed a lot of these mistakes getting repeated over time, and avoiding them is a great way to not make a fool of yourself.

Responsibility

People tend to claim that it is OK to write inefficient software, stating that computer systems are so fast these days, that you won’t see the impact. This type of mentality is dangerous in operating system design. It might be OK to write sloppy code when making a simple application, but when it comes to critical code that may get called thousands of times per second, you need to take out all the overhead you can. The operating system should supply the computer as a basic resource to the running applications, with as little complication, abstraction, and overhead as possible.

People who design operating systems in this day and age tend to have the «everything but the kitchen sink» mentality. They take it upon themselves to account for everything, which of course is good, but it shouldn’t be done at the expense of allowing poorly-written programs to flourish. There are many things that go on «under the hood» when program errors occur. Poorly-written programs cost precious execution time and involve task switches that are expensive in both memory and frequency. We encourage you to discourage poorly-written software.

Required Knowledge

If you think you can skip this, it’s just for you.

This section has been moved to a separate page because it is referred to so often in forum discussions.

Organize your plans

Before proceeding, consider what it is you want to get out of writing an operating system. What are your motivations in taking on this project? There are many possible reasons for taking on a hobby OS project, and most os-devers have more than one. Even just saying, «I just want to» can be enough, though the more you consider and clarify your goals and motives, the more you can focus on what you really want.

Be honest with yourself, as well. There’s no shame in having larger ambitions for your project, even (or especially) if they aren’t the primary objective. Try to acknowledge all of your goals, not just the one you think is your main purpose.

Try to settle on which aspects of OS design you are most interested in or see a need to work on. Most of what goes into OS dev, especially early on, is kernel design and development, but the kernel itself is only a small part of most operating systems; if your primary interest is in UX, or networking, or driver programming, you should think about whether you really need (now or in the future) to write your own OS at all or if you would be just as satisfied developing those things on an existing kernel. More than a few people have gone into OS dev when they really wanted to design a desktop environment, so this is a very important question to ask yourself.

Try to think of any non-OS projects you might want to take on first, or at the same time, especially ones which might serve as practice or preparation for the OS project. There’s usually no need to work on the OS project right now, and the more you have prepared ahead of time, the better off you’ll be (up to a point, at least—preparation is one thing, procrastination is something else).

Similarly, if you mean to work on forking an existing design to experiment with, or to modify for some specific purpose, focus on that rather than general development issues. Consider what part of the existing code base you will need, and which ones you want to change.

Try to work out some of your specific project goals, and be prepared to plan out separate projects if it helps to do so. If you are simply intending to putter around and see where it takes you, that’s fine; if your intent is to overthrow Microsoft, that’s fine (if probably unrealistic), too. Once you know what you want to do, you can break down the details of it into specific objectives, and work out what it would take to reach them. Don’t try to force too many divergent objectives into one project—if you have different things you want to try with contradictory goals, split them into different projects.

It may help if you write out an overview of your planned OS design, with any specific requirements or details you feel are notable or which could clarify what you need help with, and add it to your public repository if you can. This not only will make it easier for others to help you, it will help organize and stabilize your plans, much like writing an outline for a story or paper. Be prepared to maintain it as your goals and plans change, but keep a copy of older versions (or better still, keep the document under version control) so you can see how your work develops over time.

Finally, review the time and resources which the project will require, and decide if they are feasible. If you know that you only have a certain amount of time to devote to the project, take that into account, and whatever you do, don’t commit to an outside deadline even if you are certain you can reach it. OS dev takes time—a lot of time—and trying to finish a full OS project in a semester isn’t realistic.

Choosing your development environment

You need a platform to develop your new system on. Following the trends of general computing, the most popular is GNU/Linux, but many use Windows too. Developers using a GNU/Linux system have a slight advantage in availability of tools, but this can be solved on Windows using a system such as Cygwin or MinGW.

• Binutils: Fundamental tools for manipulation of object files.
• GCC: The GNU Compiler Collection. GCC contains compilers for C, C++, Fortran, and Ada, amongst others.
• Make: For automating the build process, which becomes really helpful once you have more than a handful of files.
• Grep and sed: For doing more powerful searches and search-and-replaces (helpful when filling out tables with data).
• Diffutils: Incredibly useful for showing the differences between two files.
• Perl or Python: One of these two scripting languages should be installed. Useful for string manipulation, amongst other things. Perl used to be the recommendation, but Python is now quite mature and is possibly easier to learn. Both have hundreds of packages/modules available for doing various tasks.
• An Assembler: For example NASM or GAS. This varies depending on your target CPU architecture.
• An editor: For writing your Assembly, C, and other (code) files.

You might not use all of these tools, but it is best to have them on hand «just in case,» and know how to use them, even at a basic level. But if you decided to use another language then the tooling is mostly up to you and maybe the list above just won’t help you in any way. Below is the information mostly related to the C/C++ or Assembly developers.

GNU/Linux

The most recommended system for OS development is GNU/Linux. When using GNU/Linux, most of the GNU development tools are probably already present. If not, use your distribution’s package management tools (APT, RPM, Portage, Pacman, Apk, etc.) to install them as needed. Again, making a cross-compiler is required, so as not to link in the development system’s runtime files.

Common editors are Vim, Emacs, KDevelop, Komodo Edit, etc. Some prefer lightweight editors instead of an IDE, such as gedit, Geany and SciTE. Many like Midnight Commander which has a Text UI and a built-in editor (mcedit) and therefore extremely lightweight and lightning fast.

About which distributions you should use, consult the list of Linux distributions. They came in every shape and sizes, and not all suited for kernel development. Do not use a distro that has a specific goal, like security (Kali, Qubes, BackTrack, Parrot etc.), science applications (eg. Scientific), firewalls and routing (eg. DD-WRT), system recovery or embedded environments (Knoppix, Rescatux, TinyCore) or specifically targeted toward beginners (like Linux Mint, Nitrux etc.) Although a beginner friendly Linux could do, choose one that’s a general purpose distro. Also use a distro which has up-to-date packages, best to pick one which uses rolling-release. Debian is easy to use, but often ships ancient, and patched versions (the tools might not behave as described). Many beginners like Ubuntu, which is fine, but it is reported to have problems with some toolchains and compilation environments (if you compile your own cross-compiler instead of using an installed one, this is a non-issue).

Best distros for kernel development are (but keep in mind this is also a matter of personal taste, so these distros are not required rather suggested, and they usually require some experience): Arch, Gentoo, Solus, Slackware, void etc. even Puppy.

If you are unsure, try Ubuntu or Manjaro.

Windows

In order to get the tools necessary, you should install the Cygwin environment. MinGW or DJGPP are alternatives, but MSYS2 is strongly suggested as it is the most complete and compatible environment, and also includes a package manager to install libraries and tools.

Microsoft has recently (as of writing) released the Windows Subsystem for Linux as an optional feature for Windows 10. It is basically a real Ubuntu command line distribution running on top of Windows WITHOUT the use of a VM. The latest GCC and Binutils (6.1.0 and 2.27 as of writing) compile and work correctly in this environment. Using the Bash shell, you can access your Windows hard disks through /mnt/ . The advantage of this solution is that you can work with whichever Windows or Linux tools that you require, without having to find out if they work in Cygwin. Many of the needed tools can be installed using «apt-get».

For all of the above, it is strongly suggested to build a cross-compiler, not only because the default compilers target different executable formats, but because it’s generally a good idea. Check the GCC Cross-Compiler page for details and instructions.

You will also need an editor. Using Notepad will work, but it’s easier if you have a more complete editor. For example, Notepad++ or Notepad2 are used by many people. If you are comfortable with Unix editors, you can pick one from the choice Cygwin provides (which includes e.g. Vim and Emacs, which take some getting-used-to but are very powerful).

It is also possible to use Visual Studio, or the freely downloadable Visual C++ Express Edition, to write and compile your operating system. You will require a special configuration file, and you will certainly be in the minority, but it does work quite well. You can even install the Windows SDK on top, enabling 64 bit development. The only pitfall is this doesn’t support Inline Assembly.

Other tools such as Watcom or Borland can be used, too, but they each have specific requirements of their own, and are not widely used for this kind of work.

MacOS

Because under the hood it uses FreeBSD’s userland, it is fully POSIX compatible. All the usual tools are available (vi, bash, dd, cat, sed, tar, cpio, etc.) Almost every tutorial works out-of-the-box. The missing tools are mostly file system related: no loopback device, no fdisk, no mkfs.vfat nor mtools. But you can use diskutil for these purposes, or use brew or macports to install those missing tools.

To get gcc, you used to have an mpkg on the 2nd Installation DVD for the older versions. Newer MacOS versions (10.13 and up) can install command line XCode (not the IDE, just the toolchain) by running «xcode-select —install» from a Terminal. This will install gcc, binutils and make. This gcc is actually a masquaraded CLang, but featurefull enough to build your own cross-compiler without problems. It is preferred to use the official compiler for bootstraping gcc than to install one from brew or macports.

Testing your operating system

The above article goes into a lot of depth about choosing how to test your operating system and how to integrate that with your development process. Both physical and emulated testing environments are discussed.

Protecting your code

During your code building you will write hundreds, even thousands, of lines of code. You’ll spend an unmentionable number of hours, and sit up late at night coding when you really should go to bed. The last thing you need is a disk crash or a poorly written ‘rm’ or ‘format’ command throwing all your work away.

What you need is a version control system. CVS has been used for a number of years, but has gotten a lot of competition from Subversion, Bazaar, Mercurial, and Git lately. If you can, you should set up a remote computer or server as a version control server, but if you do not have such a machine available you can also host the version control system on your local development computer. Just remember to backup your code to CD or FTP once in a while.

We cannot stress this point strongly enough: if you are not using source control already, you should start doing so immediately. You only need make a serious mistake in your code once to realize the importance of having your code securely versioned and easily retrievable. While it may seem like overkill for a small, private hobby project, once you get into the habit of using revision control, you’ll wonder how you ever did without it.

For Git you can create your project on GitHub. Bitbucket is also a good alternative as it supports both Git and Mercurial. Both come with free, private repositories.

An additional benefit of using version control on a network-accessible repository is that it makes it a lot easier to collaborate with and get help from others. This can be quite useful, especially in the forums, as it can avoid the need for constantly posting updated versions of your code to a message thread—you simply point the conversation towards your repository, and the others in the thread will have direct access to your most current changes. It is also crucial if, as the project grows, you begin to work with other developers on the project (just don’t expect that to happen overnight).

Common starting points

The easiest way to get a «hello world» kernel going is the Bare Bones tutorial. A different approach would be to learn how the computer itself starts up, on the Boot Sequence page.

There are many other Tutorials available, too.