Detailed Explanation of Function Return Value Mechanism in Assembly Language under Windows Environment

Return Value Rules in Windows Calling Conventions

In programming on the x86 architecture of the Windows platform, regardless of the calling convention used (<span>__cdecl</span>, <span>__stdcall</span>, <span>__fastcall</span>, etc.), the return value of a function is typically passed through the EAX register. This unified rule simplifies interoperability between different calling conventions.

Basic Return Value Mechanism

Returning Integers and Pointers

For return values of 32 bits or smaller (such as <span>int</span>, <span>short</span>, <span>char</span>, pointers, etc.), the EAX register is used directly:

; C function: int add(int a, int b);
_add:
    mov eax, [esp+4]    ; first parameter a
    add eax, [esp+8]    ; add second parameter b
    ret                 ; return value in eax

; Caller
push 10
push 20
call _add
; now eax contains 30

Returning 64-bit Integers

For 64-bit integers (<span>__int64</span> or <span>long long</span>), the EDX:EAX register pair is used:

; C function: __int64 multiply(int a, int b);
_multiply:
    mov eax, [esp+4]    ; a
    imul dword [esp+8]  ; multiply by b (result in edx:eax)
    ret                 ; high 32 bits in edx, low 32 bits in eax

; Caller
push 5
push 10
call _multiply
; now edx:eax contains 50

Floating Point Return Values

x87 Floating Point Co-Processor

Traditionally uses the x87 floating point stack’s ST(0) register:

; C function: double sqrt(double x);
_sqrt:
    fld qword [esp+4]   ; load parameter onto floating point stack
    fsqrt               ; square root
    ; result remains in ST(0)
    ret

; Caller
push dword [high_part]
push dword [low_part]
call _sqrt
; result in ST(0)

SSE Instruction Set

Modern code more frequently uses the XMM0 register:

; C function: float average(float a, float b);
_average:
    movss xmm0, [esp+4]  ; load a
    addss xmm0, [esp+8]  ; add b
    movss xmm1, [two]    ; load 2.0f
    divss xmm0, xmm1     ; divide by 2
    ; result in xmm0
    ret

section .data
two dd 2.0

; Caller
push __float32__(15.0)
push __float32__(25.0)
call _average
; result in xmm0 (20.0)

Returning Structures

Small Structures (<= 8 bytes)

Use the EDX:EAX register pair:

; C structure: struct Point { int x; int y; };
; function: struct Point get_point(void);
_get_point:
    mov eax, 100      ; x value
    mov edx, 200      ; y value
    ret

; Caller
call _get_point
; eax=100 (x), edx=200 (y)

Large Structures (> 8 bytes)

The caller allocates space and passes a hidden pointer parameter:

; C structure: struct Big { int values[8]; };
; function: struct Big make_big(void);

; Caller
sub esp, 32          ; allocate space for structure
lea eax, [esp]       ; get address of space
push eax             ; pass hidden parameter
call _make_big
add esp, 36          ; clean up stack (32+4)

; Called function
_make_big:
    mov ecx, [esp+4]  ; get hidden structure pointer
    mov dword [ecx], 1    ; initialize fields
    mov dword [ecx+4], 2
    ; ...other field initializations
    mov eax, [esp+4]  ; return structure pointer (optional)
    ret

Return Value Practices in System APIs

The Windows API strictly adheres to these rules:

; Call GetLastError()
call dword [__imp__GetLastError@0]  ; actual call to GetLastError
; error code in eax

; Call CreateFileA
push 0                  ; hTemplateFile
push 80h                ; FILE_ATTRIBUTE_NORMAL
push 2                  ; CREATE_ALWAYS
push 0                  ; lpSecurityAttributes
push 0                  ; dwShareMode
push 0C0000000h         ; GENERIC_READ|GENERIC_WRITE
push filename           ; lpFileName
call dword [__imp__CreateFileA@28]
; returned handle in eax, INVALID_HANDLE_VALUE(-1) on failure

Error Handling

Returning Error Codes

Windows API commonly uses special return values to indicate errors:

_call_api:
    xor eax, eax        ; preset 0 indicates failure
    call some_api
    test eax, eax       ; check return value
    jz .error
    ; success handling
.error:
    ; error handling

Boolean Return Values

BOOL type in Windows:

; C function: BOOL IsValidHandle(HANDLE h);
_IsValidHandle:
    cmp dword [esp+4], -1  ; compare h with INVALID_HANDLE_VALUE
    setne al               ; if not equal, al=1
    movzx eax, al          ; zero-extend to eax
    ret

Cross-Platform Comparison

While Windows primarily uses EAX, other platforms differ:

  • Linux x86: Similar to Windows, uses EAX/EDX
  • x86-64 Systems:
    • Windows: RAX/RDX/XMM0
    • Linux: RAX/RDX/XMM0/XMM1
  • ARM: R0-R3 registers

Debugging Tips

Viewing return values in Visual Studio:

  1. Set a breakpoint after the function returns
  2. View the registers window (Debug > Windows > Registers)
  3. Observe the values of EAX/RAX/XMM0

Performance Optimization Considerations

  1. Avoid returning large structures by value: Use pointer parameters instead
  2. Floating point operations: Prefer using SSE over x87
  3. Frequent calls to small functions: Use <span>__fastcall</span> convention

Summary Table

Return Value Type Registers Used Size
char/short/int EAX 32 bits
__int64 EDX:EAX 64 bits
float XMM0 32 bits
double XMM0 64 bits
Small Structures (<=8B) EDX:EAX 64 bits
Large Structures (>8B) Caller allocated memory pointer Variable

Understanding the return value passing mechanism in Windows is crucial for assembly programming and reverse engineering, as it forms the foundational protocol for inter-function communication.

Leave a Comment