Analysis of C++ Features from the Perspective of Assembly Language: Starting with the New Operator

The “Readability Crisis” of C++ Compilation Results

The assembly code generated from C++ is indeed more complex than that of C, but this complexity follows certain rules:

; C language malloc call
push    16                  ; allocate size
call    malloc
add     esp, 4

; C++ new operator call
push    16                  ; allocate size
call    ??2@YAPAXI@Z        ; operator new(uint)
add     esp, 4
mov     [ebp+var_4], eax
call    ??0MyClass@@QAE@XZ  ; constructor

Analysis of the New Operator in 32-bit Debug Driver

Basic Implementation Analysis

Given the implementation of operator new in the 32-bit debug version:

text:00011000 operator_new proc near
text:00011000 NumberOfBytes = dword ptr 8
text:11000     push    ebp
text:11001     mov     ebp, esp
text:11003     push    ecx                ; save register
text:11004     push    esi
text:11005     mov     esi, [ebp+NumberOfBytes]
text:11008     push    esi                ; size
text:11009     push    0                  ; PoolType (assumed NonPagedPool)
text:1100B     call    _ExAllocatePool@8  ; kernel allocation function
text:11010     test    eax, eax           ; check NULL
text:11012     jz      short loc_11022
text:11014     push    esi                ; size
text:11015     push    0                  ; fill value
text:11017     push    eax                ; address
text:11018     call    _memset            ; zero initialization
text:1101D     add     esp, 0Ch
text:11020     pop     esi
text:11021     pop     ecx
text:11022 loc_11022:
text:11022     pop     ebp
text:11023     retn
text:11023 operator_new endp

Key Instruction Interpretation

Function Prologue:

push ebp / mov ebp, esp: Standard stack frame establishment
push ecx / push esi: Save callee-saved registers

Memory Allocation:

push esi: Pass allocation size
push 0: PoolType parameter (debug version may not optimize constants)

Memory Initialization:

test eax, eax: Check if allocation was successful
Triplepush prepare memset parameters

Assembly Representation Patterns of C++ Features

Name Mangling

C++ uses complex name encoding to preserve type information:

??2@YAPAXI@Z => void * __cdecl operator new(unsigned int)

Decoding rules:

??2: operator new
@YA: __cdecl calling convention
PAX: return type void*
I: parameter type unsigned int

Object Lifecycle Management

Automatic insertion of constructors/destructors:

; Object creation sequence
call    ??2@YAPAXI@Z        ; operator new
test    eax, eax
jz      fail
mov     ecx, eax            ; this pointer
call    ??0MyClass@@QAE@XZ  ; constructor

; Object destruction sequence
mov     ecx, [ebp+var_4]    ; this pointer
call    ??1MyClass@@QAE@XZ  ; destructor
push    eax
call    ??3@YAXPAX@Z        ; operator delete

Comparison of Memory Allocation in C and C++

C-style allocation:

push    16
call    _malloc
add     esp, 4
mov     [ebp+ptr], eax

C++ new expression:

push    16
call    ??2@YAPAXI@Z        ; operator new
add     esp, 4
mov     [ebp+ptr], eax
mov     ecx, eax            ; this pointer preparation
call    ??0MyClass@@QAE@XZ  ; constructor

Low-Level Support for Exception Handling

The stack unwinding mechanism for C++ exception handling:

__try_block:
    push    ebp
    mov     ebp, esp
    push    -1              ; try level
    push    offset __sehtable
    push    offset __except_handler3
    mov     eax, large fs:0
    push    eax             ; install SEH
    mov     large fs:0, esp

    ; Code that may throw exceptions
    mov     ecx, [ebp+this]
    call    ?may_throw@@QAEXXZ

    ; Clean up SEH
    mov     ecx, [ebp+prev_SEH]
    mov     large fs:0, ecx
    add     esp, 0Ch
    pop     ebp
    retn

Practical: Handwritten Operator New/Delete

Optimized 64-bit Implementation

; Optimized operator new
??2@YAPEAX_K@Z proc
    mov     r8d, 'MyDr'     ; Pool label
    mov     edx, 200h       ; POOL_FLAG_NON_PAGED
    jmp     ExAllocatePool2 ; tail call optimization
??2@YAPEAX_K@Z endp

; Optimized operator delete
??3@YAXPEAX@Z proc
    test    rcx, rcx
    jz      @F
    jmp     ExFreePool      ; tail call optimization
@@: retn
??3@YAXPEAX@Z endp

Debug Enhanced Version

; Debug version operator new
debug_new proc
    push    rbx
    sub     rsp, 20h
    mov     rbx, rcx        ; save size
    lea     rcx, [rcx+10h]  ; extra allocation for debug header
    mov     edx, 200h       ; NonPagedPool
    mov     r8d, 'DbgN'
    call    ExAllocatePool2
    test    rax, rax
    jz      @F
    mov     [rax], rbx      ; store original size
    mov     qword ptr [rax+8], 0BAD0BAD0h ; debug marker
    add     rax, 10h        ; return user-available address
@@: add     rsp, 20h
    pop     rbx
    retn
debug_new endp

Understanding Advanced Techniques in C++ Assembly

Virtual Function Call Recognition:

; Virtual function call trilogy
mov     rax, [rcx]      ; get vtable pointer
mov     rax, [rax+28h]  ; get address of the 5th virtual function
call    rax             ; call virtual function

RTTI Information Location:

; Type conversion check
mov     rax, [rcx]      ; vtable pointer
mov     rax, [rax-8]    ; RTTI information
cmp     dword ptr [rax], 'MyCl'

Template Instantiation Recognition:

; Template instantiation function
??$min@H@@YAHHH@Z proc  ; int min<int>(int,int)
cmp     ecx, edx
mov     eax, edx
cmovl   eax, ecx
retn
??$min@H@@YAHHH@Z endp

By systematically analyzing these patterns, the assembly code generated by C++ will no longer be mysterious. Just like learning assembly representation for C language back in the day, the key lies in understanding how the compiler works and common idiomatic patterns. Once these rules are mastered, the efficiency of reading C++ reverse-engineered code will significantly improve.