small – UIUCTF 2018

small    |    reversing    |    250pts

This was a very fun reversing challenge that I worked on with about 5 other people. We were the first to solve this challenge. The description to this problem, which is a pretty useful hint, is that the first password is 3 characters long.Screenshot from 2018-06-19 21-43-33


The file has some invalid fields in the ELF header which make it unable to loaded into gdb.

Screenshot from 2018-06-21 20-45-58

Screenshot from 2018-06-21 20-49-37



The program asks for a password using the characters A-Z. We know from the problem description that the first password is only three characters long. The program crashes given a valid input.

I then turned to static analysis using Binary Ninja.

The main start function calls 3 functions that I have renamed.

Screenshot from 2018-06-21 21-37-44

The function display_prompt displays the password prompt ‘PWD(A-Z):’.


The function get_pw_deobfuscate_function reads user input for a password which it then uses as a repeated XOR key over the code inside obfuscated_function.

Screenshot from 2018-06-21 22-56-09

The obfuscated_function starts at 0x100C5. The memory region that gets deobfuscated starts at 0x100C6. I tried to reason about what the key could be based on the current value of the memory along with what the opcode at that address MIGHT be but this didn’t get me very far. In the meantime we ran the binary against all possible combinations because there are only 263 = 17576 combinations.

Screenshot from 2018-06-21 23-32-38

However, some of the inputs cause the program to hang because they deobfuscate the code to an operation that jumps back a few bytes only to run into the same operation again, an infinite loop… Adding a timeout fixed the problem.

Screenshot from 2018-06-22 07-34-59

The only useful input is ‘QMU’ which prints out ‘EMULATOR’ and exits.


I used Binary Ninja’s ‘XOR’ feature (Transform > XOR) with the key ‘QMU’ to view the deobfuscated function.

Screenshot from 2018-06-22 13-14-53

Deobfuscating the code also revealed a new function which simply prints the string ‘EMULATOR’.

Screenshot from 2018-06-22 13-23-20

Next, we run the file with qemu, a well known system emulator, and we see that the file is actually a bootable disk image. 

Screenshot from 2018-06-22 15-42-06

Screenshot from 2018-06-22 15-43-33

We use IDA to disassemble this bootable disk image as a binary file in 16-bit mode. We tried disassembling in 16-bit mode because we saw SeaBIOS which turned out to be a 16-bit x86 BIOS implementation. This is the BIOS that QEMU uses to load bootable images. I basically fished around for an entry point that did something that made sense. A hint probably would have been that this address corresponded to an area of code that did not disassemble properly in 32-bit x86. This revealed slightly different versions of the functions because of the instructions decoding differently in 16-bit real mode.

Screenshot from 2018-07-09 17-47-28

The password prompt is displayed using the same buffer as when run as an x86 ELF.

Screenshot from 2018-07-09 20-01-11

For some reason, I couldn’t get my laptop’s keyboard to work with qemu. I tried a USB PS/2 keyboard as well as specifying the bus/address of my keyboard however I was not able to type any input into QEMU. This could also be because the int 0x10 code corresponds to ‘set video mode’ which maybe isn’t even grabbing our input. 


Regardless, the main objective is to run the image in qemu and provide the next password to see what happens. The trick here is to patch the binary with the input we want to provide and NOP out the syscall (int 0x10) so we can execute arbitrary inputs without needing to type them in.

Screenshot from 2018-07-09 18-27-50

The pwd provided by the user is stored in the pwd_buffer. This is the same buffer that holds the pwd message so it won’t print the correct prompt but that shouldn’t matter.

Screenshot from 2018-06-22 16-08-09

Screenshot from 2018-07-09 18-17-52

At first I tried the key QMU here, and for some reason this is what we had noted down as the solution when solving the problem, however the next password is actually ‘EMULATOR’ the output when running the file as an x86 ELF with the input ‘QMU’. This makes sense because the key ‘QMU’ hints to the fact that we need to we need to run the file in QEMU and the output is the next password that we need to use.

Screenshot from 2018-07-09 18-32-00.png

Given the input ‘EMULATOR’ and booted as a disk image the file prints out the key ‘BOOTING’. I then created a version of the binary with the deobfuscated function.

Screenshot from 2018-07-09 19-39-35

I then analyzed the function in IDA as a 16-bit binary. This function seemed to print out the word ‘BOOTING’, however there also seems to be another string ‘OBSCUREDISCOVERY’ that is being operated on.

Screenshot from 2018-07-09 20-14-01.png

The in/out operations appear to read/write from a port don’t seem to to lead to anything useful and neither of these strings were the flag. My thought is that the operations on the string ‘OBSCUREDISCOVERY’ are there so that there is just some reference to this string when disassembling rather than having to manually inspect memory or run strings. We then tested both these strings as inputs to the original x86 ELF.

Screenshot from 2018-07-09 20-32-03Screenshot from 2018-07-09 20-20-11

We also tried submitting ‘OBSCUREDISCOVERY’ after providing the password ‘BOOTING’ because maybe the program was waiting for more input however that didn’t yield anything.

Screenshot from 2018-07-09 20-29-22

We then decode the obfuscated function with the XOR key ‘BOOTING’ to see what is happening since the process is hanging instead of exiting or crashing.

Screenshot from 2018-07-09 20-36-12

This function makes multiple system calls (int 0x80) to sys_socketcall which takes an argument that determines which socket function to invoke based on the following:

Screenshot from 2018-07-09 21-26-34

The first part of the function listens on a random port on localhost.

Screenshot from 2018-07-09 21-33-56

It then accepts and receives output on that port.


We use netstat to determine what port the process is listening on.

Screenshot from 2018-07-09 21-40-44

We then connect to this port via netcat and send the string ‘OBSCUREDISCOVERY’.

Screenshot from 2018-07-09 21-41-31

This gives us flag{h4ndR0113d}.

Added Bonus: Opcode not Disassembled by Binary Ninja

After I got the flag for this problem I wanted to go find the code that actually printed the flag out however I didn’t see any.

Screenshot from 2018-07-09 22-21-53

This function just stopped disassembling which was odd because the program runs cleanly to the end so the code must be executing. I was actually able to talk to the creator of this amazing problem and got to see the source for this problem. There is actually an instruction: aesdeclast xmm1, xmm2 that is being called but not being disassembled correctly be Binary Ninja. The docs show the following for this instruction:

Screenshot from 2018-07-09 22-24-49.png

We simply skip that instruction and then disassemble the rest of the code in Binary Ninja

Screenshot from 2018-07-09 22-33-00.png

The send_flag function performs the last bit of decryption and writes the flag back to our netcat connection.

Screenshot from 2018-07-09 22-46-13

Added Bonus: Keyboard Input Issue in QEMU

After talking to the creator of this problem, I found out that the keyboard inputs actually do work in QEMU by default, there is just no code in the bootable image that actually prints the input to the screen however the keyboard inputs are captured. After typing the input ‘EMULATOR’ and hitting enter the key ‘BOOTING’ appears.

Screenshot from 2018-07-23 10-40-12

Added Bonus: File running as ELF and as Bootable Image

I had the opportunity to learn about how this file is able to run as a 32-bit ELF binary as well as run as a bootable image inside QEMU from the creator of this problem. 

The 511th and 512th bytes are 0x55 and 0xAA which are the magic bytes for the first sector of a device to be a recognized as a boot sector (MBR) by most BIOSes. This works out well because the ELF headers/magic bytes start at the beginning of the file so these two don’t conflict.

When run as a bootable image, the BIOS copies the first 512 bytes (one sector) into memory at 0x7C00 and starts execution from the the first byte of data. This means that the first piece of code that is executed is actually the magic bytes in the ELF header. SeaBIOS, along with most other BIOSes, clear all the condition flags before executing the  boot code, therefore the jump branch (jg) is taken because ZF=0 and SF=OF.

Screenshot from 2018-07-23 11-08-06

Here is the file disassembled as a 16-bit image from the correct entry point.

Screenshot from 2018-07-23 11-11-40

Screenshot from 2018-07-23 11-12-28


Thanks to Benjamin Lin, the creator of this problem, for the information explained in the Bonus Section as well as sharing the source for the problem with me for learning


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s