In order to get debugging from the box we looked a little bit at the system binaries.

local_trace binary

  • receives on data on a trace socket, dumps to /tmp/local_trace.bin
  • listens on 10010 udp and receives traces
  • from what i understand:
  • there is a message queue for each process that uses dbg_trace()
  • dbg_trace() comes from libosal
  • libosal receives from message queues of the processes and sends the data via send_trace() to a local socket
  • local_trace reads from the local socket and outputs to file
  • for some unknown reason this tracing wasn't enabled and didn't work.

hooking dbg_trace()

As the functionality was already there but just not working for some reason and we were to lazy to find the reason, we just hooked dbg_trace() using LD_PRELOAD (zapinit.sh is convenient) in order to get our trace/debugging output. So we bypassed the normal tracing now by hooking the dbg_trace function, asssembling the format string on our own and passing it to another host via udp.

Compile the file using gcc -shared -ldl -fPIC libtrace.so -o local_trace.c to get a library that you can preload by replacing zapinit with a shellscript that calls zapinit and is setting LD_PRELOAD accordingly. We first dumped the trace logs using netcat. However, we had lots of packet loss while doing so. It's a mystery what the problem was, but replacing netcat with our own small toy tool (simple_udp_listen.c) solved this problem.

The complete source code: debug.tar.bz2

local_trace_config.txt

This file is stored on the device and configures the functionality of dbg_trace(). The file is opened by trace_manager() in libosal ⇒ initialize_local_trace (gets file name)

the trace file consists of:

  • arrayindex level identifying a logging facility
  • a loglevel for this facility
  • lines starting with # are skipped, just comments
  • there is a maximum number of 0x60 entries
  • levels higher than 4 are ignored, the higher the level, the more likely is the trace
  • fills osal_current_trace_level_g

So by configuring this file we can control the debugging of the device. Each individual process is checking the current log level for its facility against a certain value and if it is lower or equal calls dbg_trace(). An example:

.text:0000A7F4                 LDR     R3, =osal_trace_port_g
.text:0000A7F8                 LDR     R2, =osal_current_trace_level_g
.text:0000A7FC                 LDR     R1, [R3]
.text:0000A800                 LDR     R3, [R2,#(osal_current_trace_level_g+0x14 - 0x175F0)]
.text:0000A804                 CMP     R1, #0
.text:0000A808                 CMPNE   R3, #1
.text:0000A80C                 BLE     loc_A7D4
.text:0000A810                 BL      strerror
.text:0000A814                 MOV     R1, #5
.text:0000A818                 MOV     R2, #2
.text:0000A81C                 LDR     R3, =aFile_existsS ; "file_exists: %s"
.text:0000A820                 STR     R0, [SP,#0x68+var_68]
.text:0000A824                 MOV     R0, 0xFFFFFFFF
.text:0000A828                 BL      dbg_trace
.text:0000A82C                 B       loc_A7D4

this translates to:

r0 = errno error id (set before this snippet)
r3 = osal_trace_port_g;
r1 = *r3;
r2 = osal_current_trace_level_g;
r3 = *r2[someidx];
if(r1 != 0){
    if(r3 == 1) goto skip_debug
}
strerror(r1/errnoid);
r1 = 5;
r2 = 2;
r3 = &"file exists: %s";
esp[var_68] = r1/errnoid;
r0 = 0xFFFFFFFF;
dbg_trace

Setting these log levels to the highest possible value (4) allows to get very verbose logging. The result of this looks like:

However, we didn't get this output initially. For a reason that I explain below the logging output in the beginning was not very verbose.

There also seems to be a way to enable remote debugging by design. We didn't further look into this, but the parser checks if the buffer includes !whatever, if yes it seems to enable some remote trace option (how does it work in detail?)

  • check send_trace() for remote debugging
  • as far as i can see there is no way to enable remote debugging without patching the binary. i see no way to set the ip before its used

disabling limited trace

As said, still we ran into a problem and we saw very few actual logging happening. So I checked if the debug/tracing levels are set properly by calling dbg_get_trace_level from our hooked dbg_trace version. This showed that all trace levels were set, but those which weren't zero were one. Poking around in libosal it turns out setting the levels is done with dbg_set_trace_level/dbg_set_all_trace_levels and corresponding wrapper functions (prefixed with j_). Reading dbg_set_trace_level it was visible that there is some limited trace option, which is taken into account. If the limited tracing is enabled, all values >= 1 are set to 1 in their actual trace level. This is visible in:

In the right branch R5 would be the original R1, aka the trace level passed to this function. So we disabled the limited tracing by patching the libosal.so binary and setting this value to 0. We provide a script that takes libosal as input and patches this byte to disable the limited tracing: disable_limited_trace.py.txt

The above example of the trace output is already with this disabled.

some random notes (need a place to dump that): acquiring the called number: 5047 [9] cc_handle_SETUP: MO Call dialed number 123 needed: trace moduleid 49 (TRACE_CC), level > 1, comes from rs binary, cc_handle_SETUP, this is set as handler in cc_state_machine

Added UE state machine for TMSI/IMSI/IMEI… hooking the needed: trace moduleid 66 (TRACE_RRC), level >= 4, comes from rs, rrc_add_ue_state_machine

create_ue_sip:: Create UE: imsi (%s), msisdn (%s), rand (%s), sres (%s), kc (%s), k (%s) needed: trace moduleid 25 (TRACE_WEBC), level >= 4, comes from create_ue_ietf_sip (wsal) interestingly from what i see this is only exported, but never used. hidden cgi? strange… maybe unused

secc: Adding RAND of %s, Kc of %s and SRES of %s into database. needed: trace moduleid 32 (TRACE_SECC), level >= 4, comes from secc_database_add_subscriber(secc)

So far we are already getting RAND, Kc, SRES. Also the ipsec trace contains keys and hmac.

debug.txt · Last modified: 2011/08/31 16:09 by femto
 
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 3.0 Unported
Recent changes RSS feed Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki