Unpacking Executables - The ESP Trick
02 April 2018
By Jacob Pimental
Malware authors use many tricks to try to get past antivirus solutions. They can obfuscate strings or sign the malware as some other software. One of the more effective tricks is to use a packer to compress the malware, making it harder for antivirus software to detect it. As a malware analyst you should know about what packing is and how to unpack an executable. This article will talk about some of the basic packers and a neat trick that works to unpack most of them.
A packer is software that will compress your executable files, just like how zip files work. Normally with a zip file you have to unzip the file manually in order for it to be usable. With something called a “runtime packer” there is a portion of code in the executable that is not compressed. When you run the executable it will run the uncompressed code which unpacks the rest of the code and runs it. The reason a lot of malware authors packers is because it changes the hash value of the malware, making it hard for antivirus to identify it. We could just have antivirus detect the use of a packer, but there are other legitimate programs that use packers as well (such as video games).
How do we identify packed malware? Well one way is to look at the strings. Packed executables have very little, if any, strings that you can view. You can also look at any imports. Packed executables do not show many of their imports. This makes it hard for us to detect what a program is doing when we go to reverse engineer it. In order to successfully reverse engineer packed malware we need to debug it until we get to the decompressed memory section. Then we can dump that out and analyze that dumped executable.
One trick in doing that is the “ESP trick”. So named for the ESP register, we can use this trick to set a hardware breakpoint on the ESP register, and when we get to the breakpoint we should be at the Original Entry Point (OEP) of the program. We can then dump the rest of the executable and we should have our unpacked executable.
I made a little program to demonstrate manually unpacking executables, which you can download here. The program is a simple capture the flag challenge where you have to input a password. If you get the password correct it says “Congrats!” and if you get it wrong it says “Boo!” Simple stuff. We can easily find the password to this file by checking the strings in Radare2 or by using the “strings” command, but what if the program were packed? I am going to pack the program using the UPX packer, a free packing software that is very easy to unpack but a good learning tool nonetheless.
When we run rabin2 on our packed executable to look at imports we can see that we barely find any at all. Our strings are a mess too.
$ rabin2 -i UPX_Proj_Packed.exe
[Imports]
001 0x0040d03c NONE FUNC KERNEL32.DLL_LoadLibraryA
002 0x0040d040 NONE FUNC KERNEL32.DLL_ExitProcess
003 0x0040d044 NONE FUNC KERNEL32.DLL_GetProcAddress
004 0x0040d048 NONE FUNC KERNEL32.DLL_VirtualProtect
001 0x0040d050 NONE FUNC msvcrt.dll__iob
$
This is a clear indicator that the program is packed. We can run the program PEiD on it to see exactly what kind of packer it is using.
You can see that it is using the UPX packer in our case. This packer is extremely simple to unpack. You can actually download the UPX unpacker and unpack the program with that. I want to demonstrate unpacking the manual, and more fun, way so go ahead and boot up the program in x64dbg (x32dbg in this case as it is a 32 bit application).
The first thing I’m going to do is hit F9 (or the run button at the top) until I reach the entry point of the application. X64dbg denotes the entry point by actually showing the word “EntryPoint” next to the instruction. You can see here that we start off with the pushal instruction, a perfect identifier that the ESP Trick will work on this application.
The next step would be to take one step by hitting the F8 or F7 key, or by hitting the “step over” or “step into” buttons. Then we need to right-click the ESP register in the box on the right and click “Follow in Dump”.
Then we select the first four bytes in the dump at the bottom of x64dbg and set a hardware access breakpoint on the DWord. This will have us break right before we unpack the executable.
Now we can run our application again using the F9 key or by hitting the run button at the top. The execution should pause when we reach the hardware breakpoint we set. You can see that the instruction we break after is popal, which is a good indicator that we are on the right path. We can also see the tail jump at 0x0040c483 which is the end of the process of unpacking the executable. We need to continue running the application until we get to that point.
Once there we can step over the jmp and we should now be at our OEP.
The next step is to analyze and dump the application. Hit CTRL+A to analyze the assembly code. This just ensures that the assembly code is correct and is helpful before dumping an application. Now we can hit CTRL+I or go to Plugins > Scylla to start the dumping process.
Now click “IAT Autosearch” to automatically find the Import Address Table of the executable. After that click “Get Imports” to get a list of the imports that the executable has.
Now we can hit the Dump button and save the dumped executable wherever you want. Now we should have our dumped out unpacked executable. However, when we go to run it we get an error.
This is due to the unpacked executable not having the Import Address Table we found in our packed executable. So after we dump out the executable we then have to fix the dump. Go back to Scylla in the packed application and click “Fix Dump”. Then find that dumped executable and select it. The dump should be fixed and saved with SCY appended to the file name. We can now run the executable and see that it works.
I’m going to take this unpacked application back to rabin2 and try to dump out the strings. You can see that we actually find a lot more strings than the original packed executable, including our password string.
If we try running our application with the password “this_is_password” you can see it will work.
This is just a brief introduction to unpacking and using the ESP trick. Like with any other article, if I got something wrong or if there’s any way to improve then please tell me. I am still learning. You can contact me at my Twitter and LinkedIn.
Thanks for reading and happy reversing!
