Building a Security-Hardened VxWorks Kernel

Introduction: Why Security Hardening is Crucial

In critical environments such as defense systems, medical devices, and industrial automation, security and reliability are non-negotiable. Just as hospitals follow strict safety protocols before surgeries, real-time operating system (RTOS) developers must adhere to stringent standards to mitigate the risks of cyberattacks and human errors.

For VxWorks, Wind River provides a security-hardened kernel profile that directly maps to NIST 800-53 controls. This approach aids compliance, enhances resilience, and ensures mission-critical embedded systems are protected from vulnerabilities.

Prerequisites for Building a Hardened VxWorks Kernel

To follow this guide, you will need:

• Intel target hardware that supports TPM and secure boot (e.g., Dell Latitude E6540).
• Two USB flash drives (at least 4 GB each).
• A Windows workstation with the following software:

• Wind River VxWorks 7, SR21.07

Main References and Documentation

• Wind River

• VxWorks Hardening Guide Methodology
• VxWorks Hardening Guide

• NIST

• SP 800-53 Rev 4: Security and Privacy Controls
• Cybersecurity Framework

Understanding NIST 800-53 and Its Role in RTOS Security

NIST 800-53 defines a structured set of outcomes for secure system design. It does not prescribe strict steps but outlines the security objectives that organizations must achieve:

1. Organizational Preparedness – Coordinating processes and technologies to produce secure software.
2. Software Protection – Preventing unauthorized access to software components.
3. Producing Secure Software – Minimizing vulnerabilities in design and implementation.
4. Vulnerability Response – Quickly detecting and addressing risks post-release.

This framework serves as the foundation for the VxWorks hardening profile.

VxWorks Hardening Profile: Practical NIST Compliance

Wind River aligns VxWorks with NIST 800-53 through the hardening profile, which is built around five cybersecurity functions:

• Identify
• Protect
• Detect
• Respond
• Recover

The profile defines three levels of requirements:

• Mandatory – Must be adhered to by both Wind River and developers.
• Advisory – Optional but recommended compliance.
• Not Applicable – Not relevant to RTOS (desktop-specific).

Features of the VxWorks Hardening Profile

Upon applying the hardening profile, the following features are automatically included:

• Disk Encryption (protecting static data).
• Key Vault (secure storage for keys and credentials).
• SSH Support for secure access.
• Kernel Hardening protection.
• Secure Bootloader.
• Stack Smashing Protection in RTP.

🔐 Developers must still implement hardware security features such as secure boot, tamper protection, and patch management.

Hardened Kernel Configuration

You can build a hardened VxWorks kernel in three modes:

1. Required Controls – A baseline set of NIST 800-53 controls.
2. Required + Advisory Controls – Adds network protections.
3. Development (with shell) – For testing only; never deploy in production.

Step 1: Create a Hardened VxWorks Project

1. Open Wind River Workbench.
2. Navigate to File → New → Example → VxWorks System Setup.
3. Select VxWorks Security Hardened System.
4. Set the base name: hssHardenedVx1.
5. Choose the image: development (with shell).
6. Workbench generates:

• hssHardenedVx1_develop_vsb
• hssHardenedVx1_develop_vip
• hssHardenedVx1_develop_rtp

7. Apply security-related changes:

// In rtpPartition.c – Set default RTP path
#define EX_USE_ROMFS

// In scapVxWorks.c – Disable SCAP mechanism
#define EX_CONFIG_CHECK_SKIP

8. Update DEFAULT_BOOT_LINE with a valid IP and build the project.

Step 2: USB Setup for Hardened VxWorks Kernel

• Flash Drive 1: Copy secure boot keys (db.sig, KEK.sig, PK.sig) and bootloader files.
• Flash Drive 2: Create a 2 GB FAT partition for encrypted kernel storage.

Step 3: Configure Secure Boot in BIOS

1. Insert Flash Drive 1 and boot into BIOS.
2. Add secure boot keys (db.sig, KEK.sig, PK.sig).
3. Restart the system; VxWorks should boot with the kernel shell.

Use the devs command to identify flash drive names:

-> devs
drv refs name
  4 [ 3] /ata0a
  ...
  8 [ 3] /romfs
  1 [ 3] /ttyS0

Example: Flash Drive 1 = /bd0a, Flash Drive 2 = /bd16a.

Step 4: Staged Hardening of the Project

Recreate the project with updated device names:

• Trusted repository root → /bd0a
• Encrypted partition → /bd16a

Then repeat the USB preparation and boot steps.

At this point, you will have a development configuration of a hardened VxWorks kernel.

Next Steps: Transitioning from Development to Production

Now that you have a hardened development environment, you can:

• Build and test secure RTP applications.
• Deploy .vxe binaries to /romfs.
• Use Wind River tools for debugging.

For production deployment:

• Rebuild the project using **”Required Controls” or “Required + Advisory Controls”**.
• This will remove the kernel shell and ensure only hardened components are included.

Frequently Asked Questions (FAQ)

❓ What is NIST 800-53?

NIST 800-53 is a framework published by the National Institute of Standards and Technology. It provides security and privacy controls for federal information systems and is widely adopted in defense, aerospace, and industrial sectors.

❓ Can I use the VxWorks hardening profile on ARM-based targets?

Currently, the hardening profile only supports itl_generic BSP (Intel targets). ARM targets may not support all the secure boot and TPM features required for compliance.

❓ Why should the “Development (with shell)” kernel not be deployed?

The development configuration includes the VxWorks kernel shell, which provides powerful debugging capabilities. However, it also exposes security risks and should never be used in production systems.

❓ What security features are automatically enabled by the hardening profile?

It enables disk encryption, key vault, SSH support, kernel hardening, secure bootloader, and stack smashing protection. Developers must add hardware-level protections such as tamper resistance and patch management.

❓ Does the VxWorks hardening profile meet STIG requirements?

Yes. The VxWorks Hardening Guide is presented in STIG (Security Technical Implementation Guide) format and directly maps to NIST 800-53 controls.

Conclusion

Building a security-hardened VxWorks kernel ensures compliance with NIST 800-53 standards, enhances embedded system security, and prepares your applications for deployment in critical environments.

By following the steps outlined in this guide—covering secure boot, USB setup, BIOS configuration, and project rebuilding—you can create a solid foundation for developing secure, reliable, and resilient RTOS-based applications.

✅ Best Practice: Always use the hardening profile for production and integrate hardware-level protections early in the design process.