The purpose of writing this article is to help those who are just starting or preparing to study the Windows system. This article serves only as an experience sharing, summarizing some experiences I gathered while reverse-engineering the Windows 10 heap APIs CreateHeap, AllocateHeap, and AllocateHeapInternal. If there are errors, please correct me, seniors. We only share some common assembly feature optimizations in the Windows 10 NTDLL. For other optimizations not covered, please refer to materials for self-study.
Environment
Analysis Environment: Windows 11 Tools: Windbg, IDA 7.5 Analysis Target Platform: Windows 10 Analysis Target File: C:\Windows\SysWow64\ntdll.dll (32-bit) Version Information: 10.0.19041.5007 (WinBuild.160101.0800) SHA-1: 9C3A55D17C022D7B32EE558E8941C4C9938696CA
0
Common Release Version Optimization Summary
For common optimizations in Release version compilation, this article covers the following:
CPU Pipeline Optimization
True CPU pipeline optimization involves many complex concepts, but in code, we only need to focus on content that affects our analysis, such as out-of-order execution. Let’s take the classic three-stage pipeline as an example for easier review of pipeline optimization. The following three operations are executed synchronously by three different CPU components.
● Fetch instruction ● Decode ● Execute
In the CPU, it looks like the following process, but pipeline optimization only optimizes code that is unrelated, such as: mov eax, imm32 mov ebx, imm32, etc. If there is a sequence of instructions where the preceding and following codes are related, it will break the pipeline optimization. For a three-stage pipeline, the best case is that three lines of code are unrelated.
Below, we provide relevant illustrations to assist understanding.
Once a group of code (Group A) can form a pipeline optimization, it means that the execution order of this group of code A is unrelated to other code within the group, and this group of code may also be unrelated to other groups of code. At this point, the compiler might move this group of code forward (or backward) to execute after another group of code (Group B) without affecting the program’s logic. The simplest case is that the caller’s stack-cleaning code add esp, imm after __cdecl may occur when there is a call function for __cdecl in ntdll, which could lead to this optimization, and we just need to skip it during code restoration.
Addition Optimization (Factor Optimization)
For addition, we usually see optimizations using lea instructions, such as eax + eax + 1 in the optimized version appearing as lea eax, [eax + eax + 1], etc. This type of optimization is less common in this restoration.
1
Temporary Variable Optimization
During the reverse engineering process, we often encounter the following snippets:
In this case, when we first see it, if we find that the preceding esi originally held a variable value but was overwritten, and then after a series of operations, it was passed to a function or assigned to a variable, it may be due to the compiler optimizing out the temporary variable during the compilation process, such as the following C code.
DWORD add(DWORD a, DWORD b)
{
DWORD dwRet = a + b;
return dwRet;
}
In this case, it’s possible that the intermediate code dwRet was optimized out as a temporary variable, resulting in the following assembly form:
mov eax, [ebp + a]
add eax, [ebp + b]
When we encounter such a situation, consider that temporary variable optimization has occurred. Try to see if this register is used for other purposes. If it’s for passing parameters, it can be restored as an expression assigned to the parameter; if it’s for variable assignment, it can be expressed as assigning the expression to a variable.
2
Zero Value Register Passing
Zero value register passing was something I discovered while reverse-engineering the CreateHeap API.
Here, the EBX register is treated as a zero value throughout the function, for example, when returning a failure null value, the return value esi is set to zero.
3
Function Call Convention Optimization
This should also belong to common optimizations, but I still want to analyze it separately. For everyone accustomed to using IDA, if after pressing F5 the function parameters are incorrect, it is mostly because the compiler optimized the original __stdcall to __fastcall. For example:
When seeing the behavior of assigning values to EDX and ECX before the call, we need to enter the called function to check.
If the called function directly uses the EDX or ECX registers, consider translating that function to __stdcall or translating it as __fastcall. The specific situation should depend on readability and maintainability.
4
If Feature and Flattening Optimization (If Inversion)
Before discussing this assembly feature, let’s first look at how VS2019 optimizes the if statement as follows:
; if (a >= 0)
; {
; printf("hello world");
; }
mov edx, [a]
test edx, edx
jns xxxx ; here will be inverted
; here is the code in the statement block
push str_xxx ; HelloWorld
call subxxxxx; printf
add esp, 8; stack clean
xxxx:
; here usually for the subsequent code
Illustration:
In a normal VS2019 compiler, Release will optimize the assembly to an inverted jump form, but in ntdll, if there is a jump branch in the flattened flow segment, it needs to be translated to conditional not inverted. If the flattened flow segment immediately following the jump contains a judgment, it can also be considered to be integrated and translated into conditional expression + if statement form.
During the restoration process, there is also a case where the conditional expression is merged into the flattened flow. In this case, it also needs to be translated to conditional not inverted, and then the code in the flattened flow is translated to the conditional statement block. Generally, this situation will experience two jumps, one is jumping in, and the other is jumping out of the flattened flow. If there is an else statement, it may jump out to the front of the exit.
Let’s look at the following example, first the outer layer:
Then the inner layer:
In ntdll, we will encounter a large number of such code forms, and only in shorter conditional judgment statements can we see the shadow of normal inverted jump optimization. We need to look further back to compare whether the jump assembly labels are the same as the jumps inside the flattened flow, using context for readability judgment.
5
Push Pop Register Assignment
This situation is mostly due to insufficient registers. This situation is often followed by a call or a more complex comparison logic, as shown in the example below.
Conclusion
For reverse engineering Windows, it is essential to be familiar with the assembly code optimization features of its internal compiler to better restore it. I only posted the code I reverse-engineered for the CreateHeap API, while the code for other APIs like AllocateHeap is not shared. Note: This code is for learning purposes only, and due to project reasons, I provided a rough version of the code, and certain structures are not provided (offsets are still present). If you are interested, you can reverse-engineer and restore it yourself.
typedef struct _SYSTEM_BASIC_INFORMATION {
ULONG Reserved; // Reserved field, usually 0
ULONG TimerResolution; // System timer resolution (in microseconds)
ULONG PageSize; // Page size (in bytes)
ULONG NumberOfPhysicalPages; // Number of physical pages
ULONG LowestPhysicalPageNumber; // Lowest physical page number
ULONG HighestPhysicalPageNumber; // Highest physical page number
ULONG AllocationGranularity; // Memory allocation granularity
ULONG MinimumUserModeAddress; // Minimum address in user mode
ULONG MaximumUserModeAddress; // Maximum address in user mode
ULONG ActiveProcessorsAffinityMask; // Active processor affinity mask
UCHAR NumberOfProcessors; // Number of processors
} SYSTEM_BASIC_INFORMATION;
/*
* Flags: Specifies the optional attributes of the heap.
These options affect subsequent access to the new heap through heap functions (RtlAllocateHeap and RtlFreeHeap).
There are a total of 3 values
1. HEAP_GENERATE_EXCEPTIONS specifies that the system raises exceptions instead of returning null for exception heaps.
2. HEAP_GROWABLE indicates that the heap can grow; it must be specified if HeapBase is null.
3. HEAP_NO_SERIALIZE specifies that no mutex is used when allocating and freeing memory from this heap.
When HEAP_NO_SERIALIZE is not specified, the default is serialized access to the heap.
Serialized heap access allows two or more threads to allocate and free memory from the same heap simultaneously.
* HeapBase: If non-null, it specifies the allocated address; if null, it will be randomly allocated in the process space.
* ReserveSize:
*
*
*/
ULONG g_initVar1ByInitalizeProccess = 0;
ULONG g_initVar2ByAvrfLoadDll = 0;
typedef int(__thiscall* PFN_4B3A32F4)(DWORD, DWORD, DWORD, DWORD, DWORD, PVOID);
PFN_4B3A32F4 g_4B3A32F4;
PVOID
RtlCreateHeap(
[in] ULONG Flags,
[in, optional] PVOID HeapBase,
[in, optional] SIZE_T ReserveSize,
[in, optional] SIZE_T CommitSize,
[in, optional] PVOID Lock,
[in, optional] PRTL_HEAP_PARAMETERS Parameters
)
{
DWORD var_12C;
DWORD var_120;
DWORD var_110[10];
PVOID pLock = Lock;
ULONG ulFlags = Flags;
PVOID pBase = HeapBase;
PVOID pBase2 = HeapBase;
SIZE_T stReserver = ReserveSize;
DWORD var_BC;
SIZE_T stCommit = CommitSize;
PVOID pLock2 = Lock;
DWORD var_CC;
DWORD CriticalSectionFlag;
DWORD var_D8;
DWORD BaseAddress;
BOOL ntGlobalFlag = ((PPEB)__readfsdword(0x30))->NtGlobalFlag;
ULONG ulUnknow1 = 0;
DWORD dwRegionSize = 0;
ULONG ulUnknow2 = 0;
ULONG ulUnknow3 = 0;
DWORD pHeapHandle2;
BYTE var_A8[0x30];
DWORD dwCommiteSize;
SYSTEM_BASIC_INFORMATION SystemInformation;
DWORD var_58;
CPPEH_RECORD ms_exc;
// Simulated group
DWORD eax = 0;
DWORD ecx = 0;
DWORD edx = (DWORD)Flags;
DWORD ebx = 0;
DWORD esi = 0;
DWORD edi = (DWORD)Parameters;
// Default process heap
if (g_initVar1ByInitalizeProccess != NULL
&& pBase == NULL
&& pLock == NULL)
{
// The application is not allowed to change policies
RtlpHpAppCompatDontChangePolicy();
// If the user sets the Commit pointer, get the heap result through the Commit pointer
esi = g_4B3A32F4(ulFlags, 0, stReserver, stCommit, 0, Parameters);
if (esi != 0)
{
goto RELEASE_SRC;
}
if (edi != 0xFFFFFFFF)
{
edi = (DWORD)pBase;
esi = 0;
goto RELEASE_SRC;
}
}
else
{
if (g_initVar2ByAvrfLoadDll != 0
&& (DWORD)Parameters == 1)
{
/*
mov eax, edx
and eax, 100h ; if ((eax & 100h) != 0)
; eax = 100h
; if ((eax & 100h) == 0)
; eax = 0
neg eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF
; if ((eax & 100h) == 0)
; eax = 0 -> 0
sbb eax, eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0
not eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1 -> 0
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0 -> -1
and edi, eax ; if ((eax & 100h) != 0)
; eax = 100h -> FFFF FEFF -> -1 -> 0 -> 0
; if ((eax & 100h) == 0)
; eax = 0 -> 0 -> 0 -> -1 -> edi
*/
edi = (edx & 100) == 0 ? 0 : edi;
}
}
//edx = edx & 0xF1FFFFFF;
ulFlags = ulFlags & 0xF1FFFFFF;
if ((ulFlags & 0x100) != 0)
{
if (!(LOWORD(ulFlags) & 2)
|| (DWORD)pBase != ebx
|| ecx
|| stCommit != ebx
|| (DWORD)pLock != ebx)
{
goto RELEASE_SRC;
}
if (edi == 0xFFFFFFFF)
{
edi = !g_initVar2ByAvrfLoadDll ? 0 : edi;
}
if (edi != 0)
{
esi = edi;
// Parameter validation
if (!RtlpHpParametersVerify(edi))
{
// goto
}
edx = ulFlags;
}
else
{
esi = pHeapHandle2;
}
}
else if (_RtlpHpHeapFeatures != 1)
{
if (((((LOWORD(edx) & 2)
&& (DWORD)pBase == ebx))
// Check if heap parameters are supported
&& RtlpHpParametersSupported(edi))
|| edi == 0
)
{
eax = 2;
if ((DWORD)pLock2 == ebx)
{
esi = (DWORD)&pHeapHandle2;
}
}
}
if (esi != 0)
{
eax = (DWORD)&pHeapHandle2;
if (esi == eax)
{
memset((PBYTE)esi, 0, 0x30u);
*(WORD*)esi = 2;
*(WORD*)((BYTE*)esi + 2) = 0x30;
*(DWORD*)((BYTE*)esi + 0xC) = 1;
*(DWORD*)((BYTE*)esi + 0x10) |= 0xFFFFFFFF;
}
if ((*((BYTE*)(esi + 0x4)) & 1))
{
if (g_initVar2ByAvrfLoadDll == 0)
{
esi = 0;
goto RELEASE_SRC;
}
RtlpHpAppCompatDontChangePolicy();
esi = g_4B3A32F4(ulFlags, 0, stReserver, stCommit, 0, Parameters);
goto RELEASE_SRC;
}
else
{
eax = (DWORD)RtlpHpEnvGetEnvHandleFromParams(esi);
edi = (DWORD)stReserver;
if (edi == 0) // If Reserve is 0, provide the size of Commit
{
edi = stCommit;
}
if (eax > edi)
{
eax = edi;
}
// Create heap through RtlpHpHeapCreate and move the heap to the heap list
if (!RtlpHpHeapCreate(
RtlpHpConvertCreationFlags(ulFlags,
ntGlobalFlag),
edi,
eax,
ecx,
edx))
{
goto RELEASE_SRC;
}
// Move to the list
RtlpMoveHeapBetweenLists(esi, edx, 1, ebx);
if (*(WORD*)(esi + 0x14) == LOWORD(ebx))
{
goto RELEASE_SRC;
}
RtlpHpHeapDestroy(esi);
esi = ebx;
goto RELEASE_SRC;
}
}
if (!(ulFlags & 0x10000000))
{
if (_RtlpHeapErrorHandlerThreshold >= 2
&& (ulFlags & 0xFFF80C00))
{
if (((PPEB)__readfsdword(0x30))->Ldr)
{
//_DbgPrint();
// Normally, this should be a structure rather than a cast + offset
// But to rush the project progress, temporarily translated in the form of offset
printf("HEAP[%wZ]: ", *(DWORD*)(*(DWORD*)(__readfsdword(0x30) + 0xc) + 0xc) + 0x2c);
}
else
{
// dbg p
printf("HEAP: ");
}
// dbg p
printf("!(CheckedFlags & ~HEAP_CREATE_VALID_MASK);");
//
if (byte_4B3A5DA8 == 0)
{
RtlpReportHeapFailure(2);
}
//edx = ulFlags;
}
/*if (edx & 0xFFF80C00)
{
edx &= 0x7F3FF;
}*/
if (ulFlags & 0xFFF80C00)
{
ulFlags &= 0x7F3FF;
}
}
// ebx is always 0, here we infer var_A8 is a structure
// Perhaps initialized as STRUCT var_A8 { 0 };
// But considering WINDOWS is developed in C language,
// it is unclear what language specification version the internal Microsoft compiler supports.
// Temporarily set to memset
memset(var_A8, 0, 0x30);
if (edi != 0)
{
// Here, some settings for SEH
ms_exc.registration.TryLevel = 0;
if (edi == esi)
{
memcpy((PBYTE)edi, var_A8, 0xc);
}
ms_exc.registration.TryLevel = 0xFFFFFFFE;
}
// The assembly here is test cl, 20h but ntGlobalFlag
// is 4 bytes, I don't want to translate it to cast, I can only pray that the compiler optimizes for low bits
if (ntGlobalFlag & 0xff & 0x10)
{
ulFlags |= 0x20;
}
if (ntGlobalFlag & 0xff & 0x20)
{
ulFlags |= 0x40;
}
if (ntGlobalFlag & 0x200000)
{
ulFlags |= 0x80;
}
if (ntGlobalFlag & 0xff & 0x40)
{
ulFlags |= 0x40000000;
}
if (ntGlobalFlag & 0xff & 0x80)
{
ulFlags |= 0x20000000;
}
if (ntGlobalFlag & 0x1000)
{
ulFlags |= 0x8000000;
}
ecx = __readfsdword(0x30);
// Here, var_a8 should be a structure. Later need to change it can be inferred through PEB members
// Members of var_a8
if (!var_A8[1])
{
var_A8[1] = *(DWORD*)(ecx + 0x78);
}
if (!var_A8[2])
{
var_A8[2] = *(DWORD*)(ecx + 0x7c);
}
if (!var_A8[3])
{
var_A8[3] = *(DWORD*)(ecx + 0x84);
}
if (!var_A8[4])
{
var_A8[4] = *(DWORD*)(ecx + 0x80);
}
if (!dword_4B3A6940)
{
dword_4B3A6944 = 0x10000;
// First, query system information
if (NtQuerySystemInformation(SYSTEM_INFORMATION_CLASS::SystemBasicInformation,
&SystemInformation, 0x2c, 0) < 0)
{
goto RELEASE_SRC;
}
dword_4B3A6940 = var_58;
edx = 0x1000;
}
// Next, operations on page segments
// It seems that this is processing pages - 0x1000 operation is to set
// the previous settings as page headers
if (!var_A8[5])
{
var_A8[5] = dword_4B3A6940 - dword_4B3A6944 - 0x1000;
}
// Maximum value is 0x7F000
if (!var_A8[6] || var_A8[6] > 0x7F000)
{
var_A8[6] = 0x7F000;
}
// Added some values
if (stCommit)
{
stCommit = (stCommit + 0xFFF) & 0xFFFFF000;
}
/*edi = stReserver;
if (edi != 0)
{
ecx = edi + 0XFFF;
ecx &= 0xFFFFF000;
var_BC = ecx;
}*/
if (stReserver)
{
var_BC = (stReserver + 0xFFF) & 0xFFFFF000;
}
else
{
var_BC = (stCommit + 0xFFFF) & 0xFFFF0000;
}
/*
mov [ebp+var_BC], ecx ; ecx = var_BC
mov esi, edx ; edx = dwCommiteSize
; esi = dwCommiteSize
cmp edx, ecx
ja markif19
*/
// Calculate the maximum heap block size: the maximum heap block size is actually 0x7f000, i.e., 0x80000 minus one page.
// The maximum block size is 0xfe00, granularity offset is 3
if (stCommit > var_BC)
{
stCommit = var_BC;
esi = var_BC;
}
// Get the incoming heap parameter block, set the heap parameter block value according to the PEB,
// Set the heap block flags according to PEB->NtGlobalFlag
// ebx = 0
if (!(ulFlags & 0xff & 2) && pBase != NULL)
{
ntGlobalFlag = 0;
}
else
{
ntGlobalFlag = 0x1000;
ulUnknow1 = 2;
//eax = var_BC - 0x1000;
if (stCommit < var_BC - 0x1000)
{
var_BC = (var_BC + 0x10fff) & 0xffff0000;
}
}
if (!stCommit || !var_BC)
{
esi = 0;
goto RELEASE_SRC;
}
// Return debug heap
if (ulFlags & 0x61000000)
{
if (!(ulFlags & 0x10000000))
{
return (PVOID)RtlDebugCreateHeap(ulFlags, pBase, var_BC, stCommit, pLock2, &var_A8);
}
}
stReserver = 0x258;
if (!(ulFlags & 0xff & 1))
{
if (pLock2)
{
ulFlags |= 0x80000000;
}
CriticalSectionFlag = (DWORD)(pLock2 ? pLock2 : 0);
stReserver = !pLock2 ? 0x270 : 0x252;
}
else
{
if (pLock2 != 0)
{
esi = 0;
goto RELEASE_SRC;
}
}
// If the base address provided by the caller is 0, call ZwAllocateVirtualMemory to allocate memory from the memory manager
if (pBase == 0)
{
BaseAddress = 0;
if (var_A8[9] != (DWORD)pBase)
{
esi = 0;
goto RELEASE_SRC;
}
pLock = (PVOID)RtlpHeapGenerateRandomValue32();
var_BC = ebx;
// eax = RtlpSecMemFreeVirtualMemory(ecx, edx, &var_BC, 0x8000)
// | BaseAddress & 0x1f << 0x10;
var_D8 = RtlpHeapGenerateRandomValue32()
| BaseAddress & 0x1f << 0x10;
dwRegionSize = var_D8 + var_BC;
if (dwRegionSize < var_BC)
{
dwRegionSize = var_BC;
var_D8 = BaseAddress;
}
if (NtAllocateVirtualMemory(-1,
&BaseAddress,
0,
&dwRegionSize,
0x2000,
(ulFlags & 0x40000) == 0 ? 4 : 64) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
pHeapHandle2 = BaseAddress;
var_BC = dwRegionSize;
if (var_D8 != 0)
{
RtlpSecMemFreeVirtualMemory(var_BC, &BaseAddress, &var_D8, 0x8000);
pHeapHandle2 = BaseAddress + var_D8;
var_BC = dwRegionSize - var_D8;
}
var_CC = BaseAddress + var_D8;
stCommit = BaseAddress + var_D8;
}
else
{
if (var_A8[0] != 0)
{
if (var_A8[6] == 0 || var_A8[7] == 0 || var_A8[6] > var_A8[7] || ulFlags & 0xff != 2)
{
esi = 0;
goto RELEASE_SRC;
}
var_CC = (DWORD)pBase;
stCommit = (DWORD)pBase + (DWORD)pLock2;
var_BC = var_A8[7];
memset(pBase, 0, 0x1000);
esi = ulFlags;
}
else
{
//pBase2 needs to be switched to a structure
if (NtQueryVirtualMemory(-1, pBase, 0, &var_110, 0x1c, 0) < 0
|| var_110[0] != stCommit
|| var_110[3] == 0x10000)
{
esi = 0;
goto RELEASE_SRC;
}
// Set the heap block's reserved pages and committed pages based on the incoming ReserveSize and CommitSize
var_CC = (DWORD)var_110;
if (var_110[3] == 0x1000)
{
var_BC = var_110[2];
if (stCommit > var_BC)
{
stCommit = var_BC;
}
if (stCommit < 0x1000)
{
esi = 0;
goto RELEASE_SRC;
}
esi = ulFlags;
}
else
{
if ((ulFlags & 0x40000))
{
if (var_110[4] & 0x40)
{
goto RELEASE_SRC;
}
}
// At this point, we have obtained a heap pointer, committed base address, uncommitted base address, segment flags,
// committed size, and reserved size.
// If the committed and uncommitted base addresses are the same, we need to call ZwAllocateVirtualMemory to commit the amount specified by CommitSize
memset(var_110, 0, 0x1000);
if (NtQueryVirtualMemory(-1, pBase, 3, &var_12C, 0x1c, 0) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
var_BC = var_120;
stCommit = var_110[2];
dwCommiteSize = stCommit + var_CC;
}
}
var_110[10] |= 1;
pHeapHandle2 = (DWORD)pBase;
ulFlags &= 0x40000;
ecx = stCommit;
eax = var_CC;
}
// edi = pBase
if (var_CC == stCommit)
{
if (NtAllocateVirtualMemory(-1, &var_CC, 0, &dwCommiteSize, 0x1000, (ulFlags & 40000) == 0 ? 4 : 64) < 0)
{
esi = 0;
goto RELEASE_SRC;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
ecx = *(DWORD*)(eax + 0x50);
ecx += 0x226;
}
else
{
// Enter 0 ring from here
ecx = 0x7FFE0380;
}
if (*(BYTE*)ecx != 0)
{
if (*(BYTE*)(__readfsdword(0x30) + 0x240) != 1)
{
RtlpLogHeapCommit(pHeapHandle2, var_CC, dwCommiteSize, 1);
}
}
stCommit = stCommit + dwCommiteSize;
}
// edi = HeapHandle + 258h
if (*(DWORD*)(__readfsdword(0x30) + 0x68) != 0x800)
{
/*
;ecx = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
*(DWORD*)(pHeapHandle2 + 0xBC) = ecx;
stReserver = ecx;
edi = *(DWORD*)(pHeapHandle2 + 0xBC) + ecx;
ulFlags |= 0x4000000;
esi = ulFlags;
*/
*(DWORD*)(pHeapHandle2 + 0xBC) = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
stReserver = *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;
edi = *(DWORD*)(pHeapHandle2 + 0xBC) + *(DWORD*)(pHeapHandle2 + 0x258 + 0x7) & 0xFFFFFFF8;;
ulFlags |= 0x4000000;
esi = ulFlags;
}
var_110[9] = (stReserver + 7) & 0xFFFFFFF8;
// ecx = var_110[9] / 8;
*(WORD*)pHeapHandle2 = var_110[9] / 8;
*(WORD*)(pHeapHandle2 + 2) = 1;
*(WORD*)(pHeapHandle2 + 7) = 1;
*(DWORD*)(pHeapHandle2 + 0x60) = 0xEEFFEEFF;
*(DWORD*)(pHeapHandle2 + 0x40) = ulFlags & 0xEFFFFFFF;
*(DWORD*)(pHeapHandle2 + 0x58) = 0;
memset((BYTE*)pHeapHandle2 + 0x1F4, 0, 0x5C);
// Actually, the parameter here should be *_HEAP_HANDLE
// Here is to encrypt the heap header
RtlpCreateHeapEncoding((PVOID)pHeapHandle2);
*(DWORD*)(pHeapHandle2 + 0x234) = 1;
if (*(DWORD*)(pHeapHandle2 + 0x40) & 0x8000000)
{
// Here, perhaps to get the interceptor index
/*
Based on this guess, we provide the following definitions
typedef DWORD(__stdcall* INTERCEPTOR_PFN)(DWORD, DWORD, DWORD, DWORD);
DWORD
NTAPI
RtlpGetHeapInterceptorIndex(INTERCEPTOR_PFN);
INTERCEPTOR_PFN RtlpStackTraceDatabaseLogPrefix;
*/
/*
mov ecx, offset _RtlpStackTraceDatabaseLogPrefix@16 ; RtlpStackTraceDatabaseLogPrefix(x,x,x,x)
call _RtlpGetHeapInterceptorIndex@4 ; RtlpGetHeapInterceptorIndex(x)
movzx eax, ax; note that movzx is used, but for movzx, it takes the low bit, so we need to consider
// The return is a 4-byte structure rather than a DWORD, but we don’t know the semantics, so we give LOWORD to replace the structure
mov [esi+58h], eax
*/
*(DWORD*)(pHeapHandle2 + 0x58) = (DWORD)LOWORD(RtlpGetHeapInterceptorIndex(RtlpStackTraceDatabaseLogPrefix));
*(DWORD*)(pHeapHandle2 + 0x40) = 0xFFFFFFBF;
}
*(DWORD*)(pHeapHandle2 + 0x44) = ulFlags & 0x6001007D;
// The restoration here is questionable
/*
edi source
lea edi, [edx+258h] ; edx = heapHandle
...
mov ecx, edi ; edi = heapHandle + 0x258
mov eax, [ebp+HeapHandle]
sub ecx, eax ; ecx = 0x258
mov [eax+7Eh], cx
mov eax, [ebp+HeapHandle]
*/
*(WORD*)(pHeapHandle2 + 0x7E) = LOWORD(0x258);
// ebx is always 0
*(DWORD*)(pHeapHandle2 + 0x80) = 0;
//eax = pHeapHandle2 + 0xc0;
//*(DWORD*)(eax + 4) = eax;
*(DWORD*)(pHeapHandle2 + 0xc4) = pHeapHandle2 + 0xc0;
*(DWORD*)(pHeapHandle2 + 0xc0) = pHeapHandle2 + 0xc0;
//eax = pHeapHandle2 + 0x9c;
//*(DWORD*)(eax + 4) = eax;
//*(DWORD*)(eax) = eax;
*(DWORD*)(pHeapHandle2 + 0x9c + 4) = pHeapHandle2 + 0x9c;
*(DWORD*)(pHeapHandle2 + 0x9c) = pHeapHandle2 + 0x9c;
*(DWORD*)(pHeapHandle2 + 0xA4 + 4) = pHeapHandle2 + 0xA4;
*(DWORD*)(pHeapHandle2 + 0xA4) = pHeapHandle2 + 0xA4;
*(DWORD*)(pHeapHandle2 + 0x8C + 4) = pHeapHandle2 + 0x8C;
*(DWORD*)(pHeapHandle2 + 0x8C) = pHeapHandle2 + 0x8C;
if (!CriticalSectionFlag && !(ulFlags & 0xff & 1))
{
//CriticalSectionFlag = edi;
CriticalSectionFlag = pHeapHandle2 + 0x258;
if (RtlInitializeCriticalSectionEx(CriticalSectionFlag, 0, 0x10000000) < 0)
{
//return 0;
esi = 0;
goto RELEASE_SRC;
}
edi += 0x18;
ecx = CriticalSectionFlag;
}
*(DWORD*)(pHeapHandle2 + 0xC8) = CriticalSectionFlag;
*(DWORD*)(pHeapHandle2 + 0x48) |= 0x80000000;
if (!(RtlpInitializeHeapSegment(pHeapHandle2,
pHeapHandle2,
var_110[9] + 0x238,
CriticalSectionFlag,
var_110[0xA],
var_CC,
stCommit,
var_CC - var_110[0xE] + var_BC) & 0xff))
{
esi = 0;
goto RELEASE_SRC;
}
esi = 0x80;
if (pBase != 0)
{
memset((BYTE*)pHeapHandle2 + 0x258 + 0x18, 0, 0x80);
}
*(DWORD*)(pHeapHandle2 + 0x258 + 0x4) = 0x80;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x1c) = pHeapHandle2 + 0x258 + 0x24;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x18) = pHeapHandle2 + 0xc0;
*(DWORD*)(pHeapHandle2 + 0x258 + 0x20) = pHeapHandle2 + 0x258 + 0x24 + 0x10;
RtlpPopulateListIndex(pHeapHandle2, pHeapHandle2 + 0x258);
*(WORD*)(pHeapHandle2 + 0x7c) = 0;
*(DWORD*)(pHeapHandle2 + 0x64) = var_A8[0];
*(DWORD*)(pHeapHandle2 + 0x68) = var_A8[1];
*(DWORD*)(pHeapHandle2 + 0x6c) = var_A8[2] / 8;
*(DWORD*)(pHeapHandle2 + 0x70) = var_A8[3] / 8;
*(DWORD*)(pHeapHandle2 + 0x78) = var_A8[4] / 8;
*(DWORD*)(pHeapHandle2 + 0x5c) = (var_A8[5] + 7) / 8;
*(DWORD*)(pHeapHandle2 + 0xcc) = (var_A8[8] ^ _RtlpHeapKey);
*(DWORD*)(pHeapHandle2 + 0x250) = 4;
*(DWORD*)(pHeapHandle2 + 0x254) = 0xFE000;
if (_RtlpDisableHeapLookaside != 1)
{
//test byte ptr _RtlpDisableHeapLookaside, dl
*(DWORD*)(pHeapHandle2 + 0x48) = 1;
}
if (!(ulFlags & 0x10000))
{
*(DWORD*)(pHeapHandle2 + 0x94) = 0xf;
*(DWORD*)(pHeapHandle2 + 0x98) = 0xFFFFFFF8;
}
if (*(DWORD*)(pHeapHandle2 + 0x40) & 0x20)
{
*(DWORD*)(pHeapHandle2 + 0x94) += 8;
}
*(DWORD*)(pHeapHandle2 + 0xE4) = 0;
*(WORD*)(pHeapHandle2 + 0xE8) = 0;
*(WORD*)(pHeapHandle2 + 0xEA) = 0;
*(WORD*)(pHeapHandle2 + 0xEB) = 0;
*(DWORD*)(pHeapHandle2 + 0xE8) = 0;
// Initialize block table based on lookaside flags
if (((ulFlags & 3) == 2) && ((_RtlpDisableHeapLookaside & 1) == 0))
{
*(DWORD*)(pHeapHandle2 + 0xEC) = RtlAllocateHeap((PMY_HEAPSTRUCT)pHeapHandle2, 0x80000A, 0x100);
if (!(*(DWORD*)(pHeapHandle2 + 0xEC)))
{
goto RELEASE_SRC;
}
*(BYTE*)(pHeapHandle2 - 1) = 1;
*(BYTE*)(pHeapHandle2 + 0xf0) = 0x80;
}
// Function pointer input
/*
* Temporarily provide the following definitions
typedef DWORD(__stdcall*RtlpProcessHeapsListLock_PFN)();
DWORD
NTAPI
RtlEnterCriticalSection(RtlpProcessHeapsListLock_PFN);
RtlpProcessHeapsListLock_PFN _RtlpProcessHeapsListLock;
*/
RtlEnterCriticalSection(_RtlpProcessHeapsListLock);
// Add heap to the list
RtlpAddHeapToUnprotectedList(pHeapHandle2);
// Leave critical section
RtlLeaveCriticalSection(_RtlpProcessHeapsListLock);
if (!*(DWORD*)(pHeapHandle2 + 0x7c))
{
esi = 0;
goto RELEASE_SRC;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
eax = *(DWORD*)(eax + 0x50);
eax += 0x226;
}
else
{
eax = 0x7FFE0380;
}
if (*(BYTE*)eax != 0 && (*(BYTE*)(__readfsdword(0x30) + 0x240) & 1))
{
if (RtlGetCurrentServiceSessionId())
{
esi = *(DWORD*)(__readfsdword(0x30) + 0x50) + 0x226;
}
RtlpLogHeapCreateEvent(pHeapHandle2, ulFlags, var_BC, dwCommiteSize, (DWORD) * (BYTE*)esi);
}
else
{
esi = ulFlags;
}
if (RtlGetCurrentServiceSessionId())
{
eax = __readfsdword(0x30);
eax = *(DWORD*)(eax + 0x50);
eax += 0x230;
}
else
{
eax = 0x7FFE0388;
}
if (*(BYTE*)eax != 0)
{
// Record the created heap range
RtlpHeapLogRangeCreate(pHeapHandle2, var_BC, ulFlags);
}
*(DWORD*)(pHeapHandle2 + 0x48) &= 0x7FFFFFFF;
*(DWORD*)(pHeapHandle2 + 0xd0) = 0;
esi = pHeapHandle2; // Return value
pHeapHandle2 = 0;
CriticalSectionFlag = 0;
/*
*
Here is a unified exit for resource release and return value
The return value of this function is HeapHandle
*/
RELEASE_SRC:
// todo
edi = (DWORD)pBase;
edx = (DWORD)pLock2;
eax = ulUnknow3;
if (eax != 0
&& eax != edx)
{
RtlDeleteCriticalSection(eax);
}
if (ulUnknow2 != 0
&& edi == 0)
{
var_BC = 0;
// eax = RtlpSecMemFreeVirtualMemory(ecx, edx, &var_BC, 0x8000)
// | BaseAddress & 0x1f << 0x10;
var_D8 = RtlpSecMemFreeVirtualMemory(ecx, (PVOID)edx, &var_BC, 0x8000)
| BaseAddress & 0x1f << 0x10;
}
RET:
return (PVOID)esi;
}
Kanxue ID: TeddyBe4r
https://bbs.kanxue.com/user-home-983513.htm
*This article is an excellent article from the Kanxue Forum, authored by TeddyBe4r. Please indicate the source from the Kanxue community when reprinting.
# Previous Recommendations
1. A Preliminary Look at Current Mainstream Deobfuscation Techniques for ARM64
2. Cython Reverse Engineering in CTF Practice
3. UK NCSC: Pygmy Goat Malware Analysis Report (Third Edition)
4. Analysis Report on Anti-Cheat in Open World Adventure Games
5. Process Analysis of Travel APP (TCP + protobuf)
Share the Ball
Like the Ball
Watch the Ball
Click to read the original text for more information