Detailed Explanation of x86 Assembly Language Instruction Encoding

Basics of Instruction Encoding

The process of translating assembly language instructions into machine language is called instruction encoding, while the process of converting machine instructions back to assembly language is called instruction decoding. The Intel x86 architecture employs a CISC (Complex Instruction Set Computer) design, making its instruction encoding system relatively complex yet flexible.

x86 Instruction Format Structure

x86 machine instructions use a modular format, which includes the following optional fields:

[Prefix] [Opcode] [ModR/M] [SIB] [Displacement] [Immediate]

1. Instruction Prefix Byte

• Overrides the default operand size
• Includes segment override prefix, operand size prefix, address size prefix, etc.
• Located at the start address of the instruction

2. Opcode Field

• Specifies the specific variant of the instruction
• For example, the ADD instruction has 9 different opcodes depending on the type of parameters
• This is the only mandatory field

3. Mod R/M Byte

The structure of the Mod R/M byte is as follows:

7   6   5   4   3   2   1   0
+---+---+---+---+---+---+---+---+
| Mod | Reg/Opcode | R/M        |
+---+---+---+---+---+---+---+---+

Meaning of the Mod Field:

Mod	Displacement Description
00	DISP=0, no displacement (unless r/m=110)
01	DISP=8-bit displacement, sign-extended to 16/32 bits
10	DISP=16/32-bit displacement
11	The R/M field contains the register number

16-bit Mode R/M Field (when Mod=10b):

R/M	Effective Address	R/M	Effective Address
000	[BX+SI]+D16	100	[SI]+D16
001	[BX+DI]+D16	101	[DI]+D16
010	[BP+SI]+D16	110	[BP]+D16
011	[BP+DI]+D16	111	[BX]+D16

Single Byte Instructions and Immediate Operands

Single Byte Instructions

Some simple instructions consist of a single byte opcode:

; Single byte instruction examples
NOP        ; 90h - No operation
CLC        ; F8h - Clear carry flag
STC        ; F9h - Set carry flag
CLI        ; FAh - Clear interrupt flag
STI        ; FBh - Set interrupt flag

Immediate Operand Encoding

Immediate operands are directly embedded in the instruction, immediately following the opcode:

; Immediate operand examples
MOV AL, 5        ; B0 05
MOV AX, 100      ; B8 0064
MOV EAX, 1234h   ; B8 34120000
ADD AX, 10       ; 83 C0 0A

Characteristics of Immediate Encoding:

• Stored in little-endian format (least significant byte first)
• Size depends on the operand size attribute
• Can be 8-bit, 16-bit, or 32-bit

Masked and Unmasked Exceptions

Exception Classification

Exceptions in the x86 architecture are classified into masked and unmasked:

Unmasked Exceptions

• Cannot be masked by program control
• Execution must continue after handling
• Examples:
• Division error (exception 0)
• Invalid opcode (exception 6)
• General protection fault (exception 13)
• Page fault (exception 14)

Masked Exceptions

• Can be controlled by flags to determine whether to trigger
• Main Types:

1. Floating Point Exceptions

Masked via the control word register:

; Set floating point control word
FSTCW [control_word]
AND WORD [control_word], 0FFC0h  ; Mask all floating point exceptions
FLDCW [control_word]

2. Single Step Exceptions (Traps)

Controlled via the TF bit of EFLAGS:

PUSHF
AND WORD [SP], 0FEFFh  ; Clear TF bit (disable single step)
POPF

3. Breakpoint Exceptions

Controlled via debug registers.

Exception Handling Mechanism

; Exception handling example
divide_error_handler:
    ; Handle division error
    IRET

; Set interrupt descriptor table
MOV AX, 0
MOV DS, AX
MOV WORD [0*4], divide_error_handler  ; Set division error handler
MOV WORD [0*4+2], CS

Practical Encoding Examples

Example 1: Register to Register Transfer

MOV AX, BX  ; Encoding: 89 D8
; Analysis:
; Opcode: 89 = MOV r/m16, r16
; ModR/M: D8 = 11 (Mod) 011 (BX) 000 (AX)

Example 2: Memory Addressing

MOV [BX+SI+10h], AX  ; Encoding: 89 40 10
; Analysis:
; Opcode: 89 = MOV r/m16, r16
; ModR/M: 40 = 01 (Mod) 000 (AX) 000 ([BX+SI]+disp8)
; Displacement: 10 = 16-byte displacement

Example 3: Immediate to Register

MOV CX, 1234h  ; Encoding: B9 3412
; Analysis:
; Opcode: B8+ reg = MOV reg16, imm16
; CX register encoding is 001, thus opcode=B9
; Immediate: 3412 (little-endian)

Conclusion

x86 instruction encoding reflects the complexity of the CISC architecture:

• Flexibility: Supports various addressing modes and operand types
• Modularity: Achieves rich functionality through the combination of different fields
• Compatibility: Maintains backward compatibility, supporting evolution from 16-bit to 64-bit

Understanding instruction encoding is crucial for mastering the x86 architecture, performing disassembly, and optimizing code. Although modern compilers can automatically handle these details, understanding the underlying mechanisms remains key to becoming an advanced programmer.