The Evolution of macOS: A Comprehensive Overview

macOS, as the designated operating system for iOS development, is frequently interacted with since iOS itself is derived from it. To test your familiarity with it, see if you can answer these questions:

• When we talk about MacOS, we usually include “X” and call it MacOS X or OS X. Why is that? What does “X” mean?
• What do Darwin, XNU, Mach, and BSD represent, and what is their relationship?
• What is the relationship between Mac OS and Unix?
• Apple open-sourced Darwin, why did the corresponding open-source community fail to develop?

This article aims to help answer these questions and will also share some background stories related to Apple.

The content of this article is mainly referenced from “Mac OS X And iOS Internals” (the Chinese version is called “深入解析Mac OS & iOS”) and “*OS Internal Volume I — User Mode”, both authored by Jonathan Levin. The former was completed in 2012, while the latter’s second edition was completed in 2019. The latter is not only a refined version of the former but also a completely new edition, with a lot of illustrations and texts rewritten. As time goes by, the latter has had more discussions on the latest technologies.

Background of macOS Development

Background

The early version of macOS was called Mac OS Classic, which was born at Apple, featuring a great GUI but was a relatively rough and immature operating system.

During this period, Apple’s founder Steve Jobs left Apple to establish NeXT, which produced NeXT computers and NeXTstation, running on an operating system called NeXTSTEP. NeXTSTEP had these avant-garde features:

• Utilized Mach microkernel
• Used Objective-C as the development language
• Object-oriented principles permeated the entire operating system
• Device driver development was an object-oriented framework called DriverKit

Birth of MacOS X

Later, Jobs returned to Apple and brought NeXTSTEP back with him. Naturally, Mac OS Classic and NeXTSTEP, two very niche operating systems, merged. One had a great GUI but poor design, while the other had a great design but a bland GUI. The combination resulted in a popular operating system, MacOS X. The “X” means the Roman numeral “10”, corresponding to the version number of MacOS at that time, 10.x, which has since developed for a long time.

Several core components of MacOS X at this time: Cocoa, Mach, IOKit, and Xcode’s Interface Builder all came from NeXTSTEP. The kernel of this operating system is Darwin.

Darwin is open-source, and a series of variants such as iOS, tvOS, watchOS, and BridgeOS (the OS for the MacBook Touch Bar) were born from it. There is a command to check the Darwin version information used by the system: uname.

$ uname -v
Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64

Versions of Darwin and its variants are updated synchronously. The version relationship between them follows this:

Darwinver = 10.(MacOSVer + 4) = (iOSVer + 6) = (TvOSVer + 6) = (WatchOSVer + 13)

Later, the main version number of MacOS was upgraded from 10 to 11, and the version correspondence of MacOS changed slightly. Here, it is necessary to review the naming changes of MacOS, as very few people can accurately call it due to its numerous naming changes.

To avoid confusion, from this point on, when referring to MacOS, it will be referred to as macOS.

Evolution History of the Darwin Operating System.

Image from (*OS Volume 1)

Combining the above image, we can tell a small story about iOS. The initial codename for iOS1.x was Alpine, which was the default root password for i series devices. But the final released version was codenamed Heavenly, because this version of the operating system had complete debugging symbols, was unencrypted, and easy to disassemble. Many jailbreakers relied on the symbols and function call relationships extracted from this version for inspiration for cracking. From the jailbreakers’ perspective, it was indeed as beautiful as heaven.

Internal Composition of Darwin

Darwin is a UNIX-like core operating system, which can be roughly seen as: Darwin = kernel + XNU + runtime. macOS has been a certified UNIX implementation since Leopard (10.5).

XNU is a key component of Darwin, where XNU = Mach + BSD + libkern + IOKit.

The original version of XNU was the core of NeXTSTEP, which included Mach version 2.5 and BSD version 4.3. After NeXTSTEP was integrated into Apple, Mach was upgraded to 3.0, and BSD was upgraded to FreeBSD.

Mach and BSD are a microkernel and a monolithic kernel, respectively, so XNU is a hybrid architecture. The key to understanding these types of kernels is to pay attention to the range occupied by kernel mode and user mode.

Mach

Mach (microkernel) was developed by Carnegie Mellon University, aiming to replace the BSD UNIX core. This microkernel can only handle the most basic operating system responsibilities:

• Process and thread abstraction
• Virtual memory management
• Task scheduling
• Interprocess communication and messaging mechanism

As shown in the above image, the functionality of the microkernel is inherently limited, and other OS functionalities are built as basic services in user mode. Because of this characteristic, the internal task calls will have more frequent context switches between kernel mode and user mode, which will consume extra time. Meanwhile, the messaging between the kernel and service processes will also reduce running efficiency, so this design usually lowers performance.

However, it also has advantages, such as easy extensibility of service processes, and problems with service processes will not jeopardize the kernel. Thanks to this extensibility, MachO can support multi-architecture files, allowing macOS to smoothly transition from PowerPC to Intel and then to M1.

BSD

BSD (monolithic kernel) is an abbreviation for Berkeley Software Distribution, a derivative of Unix. BSD exists to improve Mach, building on top of Mach and providing a more reliable and modern API. It mainly includes:

• UNIX process model
• POSIX thread model
• UNIX users and groups
• Network protocol stack (BSD Socket API)

The characteristic of a monolithic kernel is that both user services and kernel services run in the same memory space, which effectively reduces frequent switching between kernel mode and user mode, thus improving execution efficiency. However, monolithic kernels are not without drawbacks; they have poorer extensibility, and if one service in the kernel crashes, the entire operating system will crash.

Darwin Architecture

Since there is no perfect kernel mode, Apple combined the two, balancing the advantages of both microkernels and monolithic kernels, which is Darwin.

Image from (Mac OS X And iOS Internals)

Here, XNU is not represented; its boundary can be seen as the Kernel/User Transition, below which includes the hierarchy of BSD and Mach, which is XNU. In the macOS system, the layers above Darwin are mostly not open-source and are Apple’s proprietary property.

XNU also has two other important components:

• libkern: This is a built-in C++ library that supports C++ runtime. With it, many advanced features of the kernel can be implemented in C++.
• IOKit: This is a device driver framework that, with the underlying support provided by libkern, allows drivers to be implemented using C++. Thanks to C++’s object-oriented features, external developers save a lot of costs when creating drivers.

The Open Source Journey of Darwin

Since Darwin is open-source, why haven’t any non-Apple Darwin distribution operating systems emerged? Although it is open-source, the Mach and BSD that XNU mainly relies on are originally open-source; Apple also closed the support for ARMv7/8 separately; the original launchd source, after Mac OS 10.10, also became a closed-source project incorporated into the libxpc project. Not to mention, the open-source version of Darwin has not been completely stripped of Apple’s private APIs, making it unable to compile completely and requiring some additional modifications.

There are two important open-source versions surrounding Darwin: OpenDarwin and PureDarwin, which we can look at their current development status.

OpenDarwin

It was established by Apple in April 2002, aiming to enhance collaboration between Apple developers and the free software community and to develop Darwin into another independent version. Ideally, Apple could apply improvements from OpenDarwin to Darwin, while the open-source design could fully control the system for use in distributions like GNU-Darwin. However, only four years later, OpenDarwin announced its closure. The following is the statement from the OpenDarwin project team:

Over the past few years, OpenDarwin has become a mere hosting facility for Mac OS X related projects. The original notions of developing the Mac OS X and Darwin sources have not panned out. Availability of sources, interaction with Apple representatives, difficulty building and tracking sources, and a lack of interest from the community have all contributed to this. Administering a system to host other people’s projects is not what the remaining OpenDarwin contributors had signed up for and have been doing this thankless task far longer than they expected. It is time for OpenDarwin to go dark.

The main factors are two:

• Apple’s macOS X has too strong control over OpenDarwin, preventing its independent development
• The open-source community’s interest has waned, which could be said to be caused by the former

Currently, the OpenDarwin website http://opendarwin.org/ only has one line: Opendarwin memorial page.

PureDarwin

PureDarwin is generally regarded as the successor to OpenDarwin. Its code is hosted on Github and is still maintained. The meaning of “Pure” is cleaner; PureDarwin only uses components released by Apple for Darwin without using other components of macOS. Its goal is to make Darwin easier to use by providing documentation that allows open-source enthusiasts and developers to retrieve, understand, modify, build, and distribute Darwin.

Due to the lack of official support, it currently has only 1.7k stars on Github, indicating that its attention and development are not very good. However, when asked why they spend time maintaining PureDarwin, their answer is:

For learning and for fun.

Short yet uplifting, while also carrying a hint of sadness.

So, returning to the previous question of why the open-source Darwin has not developed, it is because it is an operating system created for macOS, relying on the features of macOS and Apple’s support. Attempting to walk the “Pure” Darwin open-source route without these two is very difficult.

Hackintosh

The development of OpenDarwin and PureDarwin has also led to some beneficial things, as the open-source Darwin was made into a complete bootable and installable ISO image. Subsequently, the OSX86 project (which aims to port Apple’s macOS operating system to non-Apple computers) continued to promote this, striving to fully port macOS to PCs and laptops, a practice known as Hackintosh.

The usual Hackintosh solutions are achieved through bootloaders that do not modify the macOS source files, considered the best legal way. Apple once open-sourced Boot-132, a bootloader for loading the XNU kernel. The Voodoo team developed the Chameleon bootloader based on this program, and later Clover appeared, allowing computers that do not support EFI to enter a simulated EFI environment. Now there is OpenCore, which is more complex in configuration but has gained a considerable number of users due to its compatibility with many kext authors and its ease of use.

Regarding compliance issues, although the boot method does not modify the macOS source code, Apple’s end-user license agreement (EULA) does not allow macOS to be installed on hardware without the Apple trademark. Apple has sued and won several commercial actions related to Hackintosh, but has not paid much attention to non-profit individual Hackintosh activities.

The Story of the Mascot

People are keen to set mascots for popular operating systems or frameworks. The mascot of Linux is a penguin (named Tux, short for Torvalds UniX), and the mascot of Android is a green little robot (no official name, referred to by developers as Bugdroid). Now let’s take a look at the mascots of two operating systems related to macOS.

BSD

The mascot of BSD is a little devil 😈, called Beastie, which sounds very similar to BSD. It usually carries a trident, representing the fork of a journey.

Darwin

The mascot of Darwin is Hexley, a cartoon platypus wearing a BSD little devil hat and holding a trident. Hexley was designed by Jon Hooper, and the copyright belongs to him. However, Hexley does not belong to Apple. Originally, the mascot’s name should have been Huxley, named after the British biologist Thomas Henry Huxley, who defended Darwin’s theory of evolution. The original proposer mistakenly thought it was Darwin’s assistant and misused the name Hexley. By the time the mistake was discovered, it was too late to change the name, so Hexley remained.

This image does not belong to Apple but to the open-source community, so the open-source version of Darwin displays this pattern.

The cover of the “*OS Internal Trilogy” book series uses the Hexley image.

Future Outlook

The outlook for the future of macOS is drawn from “Mac OS X And iOS Internals” (note that its completion time is 2012). Looking back ten years later, we can see how these predictions have been realized.

Eradicate Mach

The Mach API in the kernel is a product of the NeXTSTEP era, running slowly and having difficulty matching the execution efficiency of BSD. Moreover, XNU itself tends to a monolithic kernel architecture. If Mach were removed and the kernel built entirely on BSD, there would be great benefits, but this indeed requires a huge amount of work.

This idea has not been achieved, and Apple has no plans to do so; the hybrid kernel will continue to exist for a long time.

Compatibility with ELF Format

The biggest difficulty for macOS to integrate into the UN*X world is its insistence on using the Mach-O binary format. Of course, this idea also relies on the previous eradication of Mach, so that programs in Linux and BSD can be migrated to macOS without modification.

This is a beautiful imagination; considering Apple’s crackdown on Hackintosh, its business strategy is to monopolize and fully control rather than expand market share.

Use of ZFS

The file system used by macOS in its early days was HFS+, but this file system has faced a lot of criticism. Linus once commented on HFS+:

Quite frankly, HFS+ is probably the worst filesystem ever. Christ what shit it is.

HFS+ indeed has many shortcomings; it is case insensitive, does not support checksum verification of data content, and timestamps only support up to second-level precision. At that time, Sun developed ZFS, which was claimed to be the “most powerful” file system in the universe. There were rumors that Apple would use this file system, but later Sun was acquired by Oracle, and this idea ultimately could not be realized.

In 2017, with the release of the Mac OS High Sierra version, Apple officially launched the Apple File System (APFS). This file system was developed from scratch by Apple, taking three years. For a complete file system, this efficiency is already very high. It supports more features and claims to have optimized for SSDs. However, it is regrettable that APFS’s performance has not exceeded that of HFS+ in several aspects.

Merging with iOS

During the development of macOS and iOS, many features were migrated from one platform to another after maturing on one platform. Even before the M1 chip appeared, this idea was already possible; Apple had previously implemented a hardware architecture translation mechanism — Rosetta.

After the release of the M1 chip, both macOS and iOS can run on ARM64 architecture chips, making this idea seem natural to implement. However, reality is not so; compared to unification, Apple prefers each product line, macOS, iPadOS, and iOS, to have its applicable scenarios, and Apple vigorously promotes this, which helps with product sales.

Conclusion

To review the content of this article, here are the questions mentioned at the beginning:

1. When we talk about MacOS, we usually include “X” and call it MacOS X or OS X. Why is that? What does “X” mean?

2. What do Darwin, XNU, Mach, and BSD represent, and what is their relationship?

3. What is the relationship between Mac OS and Unix?

4. Apple open-sourced Darwin; why did the corresponding open-source community fail to develop?

References