In this experiment, we will analyze a specially prepared binary file (non-malicious), which employs various anti-decompilation techniques.

First, open antidisasm.exe using IDA:

You can see a set of function calls here, each using different anti-decompilation techniques, storing the return values in the eax register. The task is to find out the return value of each function through static analysis.

1) Analyze Loc_40101A Function

First, enter the function at 0x40101A

It appears that IDA does not recognize this as a function, and it seems this function has no return value because there are some garbage instructions after the CALL EAX instruction, and the loc_401045 label marks the start of the next function.

Note that there is a strange call (call $+5) at the start of loc_40101A.

This interesting CALL’s address is the next instruction (0x401022). The purpose of this call is to push the return address (0x401022) onto the stack, and then load that address into EAX by executing the POP EAX instruction.

Then EAX is incremented by 10, and a CALL is made to this new value of EAX.

Now you know EAX’s value is 0x401032 (0x401022 + 0x10). Unfortunately, this address is located in the middle of garbage code, and it seems there are no instructions at this address.

It is clear that this garbage code is due to the disassembly engine misaligning while parsing certain instructions. When IDA parsed the CALL EAX instruction, it did not know what address it was calling, so it attempted to disassemble the next instruction after CALL EAX.

To fix this issue, first select all the garbage code, then right-click and choose undefined (or press U):

Next, select the first byte at the location 0x401032, press C to convert the byte at this location into code. You can see that after the CALL EAX instruction, a string “Fintastic!” appears.

Now the code is much clearer, and you can see that the return value of loc_40101A function is 0x1337.

To summarize, in this function, we learned two anti-decompilation techniques: first, there is an indirect call to a dynamically calculated address. IDA does not know what the called address is, so it tries to parse the string embedded in the CALL instruction (the second anti-decompilation technique). Ultimately, this results in parsing out some garbage instructions instead of a valid assembly instruction.

2) Analyze loc_401045 Function

This time we will analyze the function located at loc_401045.

At first glance, although IDA does not recognize the code here as a normal function, there is a classic function start and end retn instruction. You can also highlight the EAX register to find where it is assigned.

It seems that EAX is first assigned to 0x11EB, then increased by 0x1000. Even so, we should focus on the jump instruction (jz), which appears to jump to the middle of another assembly instruction. Also, take note of the red cross reference here, which seems to indicate some issue.

Before analyzing where jz will jump to, let’s first see what condition it will jump on. The last instruction that sets the zero flag ZF to 1 is the XOR EAX, EAX instruction before the jz jump instruction, which clears the EAX register and sets the ZF flag to 1. This indicates that the jump will definitely be executed.

Since it jumps to the middle of an assembly instruction, select this instruction and convert it to data (using Undefine or press U).

IDA may convert more code than you expect, but that’s okay, because you know that the target address of the jz jump is 0x40104B, and the original jz instruction is at 0x401050.

Now select a byte at 0x40104B, press C to convert it to code, and do the same operation to convert the jz instruction at 0x401050. After these operations, you should see code like the following:

This indicates that another jump instruction jmp is hidden in the middle of the PUSH instruction.

As you can see, the target address of this hidden jump instruction is 0x40105E (the middle of another assembly instruction 0x40105D). However, this time it appears to be a normal assembly misalignment parsing instruction.

To continue, convert 0x40105D to data, then convert the address at 0x40105E to code. After conversion, the code you see should look like this:

Now you can clearly see that the return value is set to 0x4096. Note that the garbage bytes after the return instruction retn were previously used to prevent IDA from disassembling the instruction assigning to EAX.

The following screenshot shows the execution flow of this function before any modifications:

To summarize, through this function, you learned some anti-decompilation techniques. The most obvious one is jumping to the middle of another instruction. In this case, the PUSH instruction was used to hide another jump instruction. You also saw the conversion of a jz conditional jump into an unconditional jump, as well as some garbage bytes to prevent IDA from disassembling.

3) Analyze sub_401065 Function

The next function to analyze is sub_401065. However, this time IDA recognized it as a normal function.

You can see that EAX is first cleared, then the sub_40107D function is called (with parameter 0x1000), and eventually, EAX is increased by 0x1000. The question is whether the return value of the sub_40107D function modifies EAX.

Let’s take a look at the sub_40107D function:

It seems that this function does nothing but assign the parameter 0x1000 to EAX and then adds 0x1000. When this function returns, EAX should be 0x2000. Does this mean that the return value of the sub_401065 function is 0x3000 (0x2000 + 0x1000)?

As you might suspect, it is not that simple. Let’s see what happens before sub_40107D returns:

First, the address of the first parameter is passed to EDX, and then EDX is decremented by 4. Do you know where EDX is pointing now? Let’s take a look at the stack structure:

After decrementing EDX by 4, it points to the return address on the stack. Then, in the third line, the return address is increased by 0x2B. This indicates that the return address of the current function sub_40107D has been modified, and it will jump to another place in the code.

To check where it will jump to, return to sub_401065:

The return address is 0x401074. However, you know it has been incremented by 0x2B. So sub_40107D will return to 0x40109F (0x401074 + 0x2B). Switch to the next view to find this return address.

Not surprisingly, the location of the return address has garbage instructions. Press U to convert it to data, then at 0x40109F, press C to convert it to code:

We see that the final value of EAX is 0xC0DE!

To summarize, through the analysis just now, we learned a popular anti-decompilation technique called return address replacement. Malicious code misled IDA by replacing the return address in the calling function, causing IDA to interpret the return address incorrectly.

4) Analyze sub_4010B2 Function

Now let’s look at the incomplete sub_4010B2 function.

Middle code ignored

If you enter this function and view the disassembly code, you will see that many disassembly codes are using various instructions to manipulate the EAX register. Despite this, if you look closely, you will find that some combinations of instructions are performing meaningless operations.

This is a relatively simple obfuscation technique. These garbage instructions embedded in normal code not only make manual analysis more difficult, but also do not affect the execution flow of the program.

The only way to deal with these garbage instructions is to find a pattern among them. If you find the pattern of the garbage instructions, you can try to write a script to overwrite these garbage instructions with NOP instructions or use some color to highlight them. However, writing scripts is beyond the scope of this tutorial.

If you analyze these garbage instructions closely, you will find that only three instructions affect the final value of EAX:

This indicates that the final value of EAX is 0x1500.

5) Analyze sub_40116D Function

The last function to analyze is sub_40116D.

In this program, the return value of EAX appears to be 0xEBFE. Even so, you should immediately notice:

The instruction MOV FS:0,ESP indicates that it is installing a Structured Exception Handler (SEH).

The exception handler is saved as a singly linked list structure, where each node is an EXCEPTION_REGISTRATION structure:

_EXCEPTION_REGISTRATION struc

prev     dd ?

handler dd ?

_EXCEPTION_REGISTRATION   ends

This structure consists of two fields: the first field prev is a pointer to the next EXCEPTION_REGISTRATION structure, and the second field handler points to the exception handling function.

The pointer to the first EXCEPTION_REGISTRATION structure is always stored in the first DWORD byte of the TIB. On Win32 platforms, the address of the TIB is stored in the FS register, so the instruction MOV FS:0,ESP effectively sets the saved value at the top of the stack as the first exception handling structure EXCEPTION_REGISTRATION.

In the sub_40116D function, the stack structure should look like this (after installing SEH):

So the question arises, is there an exception triggered in this function? The answer is yes. First, look at the ECX register, which is set to 0, and then the program attempts to write a DWORD to the address saved in ECX. Since ECX points to an unallocated memory address 0x00000000, this will definitely trigger an illegal memory access exception (STATUS_ACCESS_VIOLATION-0xC0000005).

But what is the address of the exception handling function? In this example, you see 0x15232A1 pushed onto the stack, but it is not a valid function address. Note the XOR instruction in the above image, which XORs 0x1122300 with the address of the exception handling function. This indicates that the real address is 0x15232A1 xor 0x1122300 = 0x4011A1.

You can also use the calculator in IDA to calculate it (View->Calculator or press SHIFT+/):

Now switch the view to the address at 0x4011A1:

Repeat the previous routine, convert the data at 0x4011A1 to code:

You can see that EAX is assigned the value 0x512, while other instructions merely restore the stack frame and jump to the end of the sub_40116D function.

To summarize, in this function, we learned to use SEH to alter the program’s execution flow. SEH is often used for anti-decompilation and anti-debugging techniques. Additionally, the address of the exception handler was also obfuscated using XOR.

– End –

Kanxue ID: Adventure

https://bbs.pediy.com/user-749942.htm

This article is originally by Adventure on Kanxue Forum

Reprint must indicate the source from Kanxue Community

Recommended Books:

ClickBuy Now!

Recommended Popular Technical Articles:

• Analysis of Seafile Cloud Storage’s End-to-End Encryption

• VMP3.2 License Analysis

• Introduction to Reflection DLL Technology

• X86 Assembly: Is It Really a “Terrifying” Existence?

Official WeChat ID: ikanxue Official Weibo: Kanxue Security

Business Cooperation: [email protected]