Debugging Tools and Techniques

This is the html version of the file http://thermalnoise.files.wordpress.com/2007/01/debugging.pdf.

Google automatically generates html versions of documents as we crawl the web.

Page 1

Debugging Tools and Techniques

Thus Spake The Master Programmer: “When

you have learned to snatch the error code from

the trap frame, it will be time for you to

Leave.”

- The Tao Of Programming

Ananth Shrinivas

Solaris Engineering

Sun Microsystems

Page 2

Approx & Non-Linear Agenda

➢ Common Problems and Common Tools

➢ The Problems in Theory

➢ Memory Management

➢ Profiling and Execution Tracing

➢ Core Dumps, Network Monitoring

➢ The Solutions in Practice

➢ GDB, Netcat, Wireshark (The Swiss Army Knives)

➢ strace and ltrace (Dynamic Execution Tracers)

➢ Valgrind and Friends (Emulators and Interposing Libraries)

➢ gprof and oprofile (Instrumentation and Sampling Profilers)

➢ The One True Tool: State of Art in Debugging

Page 3

Common Problems

➢ Memory Management: Invalid Pointers, Buffer

Overflows, Double Frees, Memory Leaks

➢ Tracing Execution and Flow Control: Profiling

and Performance analysis, Code path verification,

Code coverage, Debugging Logical errors

➢ Multi-threaded Programming: Race Conditions,

Deadlocks, Lock Contention

➢ Advanced Problems: Core Dumps, Disassembly,

Debugging your operating system, Compiler Bugs,

Hardware Bugs, Debugger Bug, Understanding

foreign code !

Page 4

Common Linux Debugging

Tools

➢ Memory: valgrind, Insure++, Purify, memwatch

➢ Execution Tracing: strace, ltrace, gdb

➢ Process Monitoring: pmap, lsof, top, /proc

➢ Profiling: gprof, oprofile, CodeAnalyst, vTune

➢ Code Coverage: gcov

➢ Multithreaded Programming: helgrind, $BRAIN

➢ General Purpose Debuggers: gdb, dbx, DDD

➢ Static Code Analyzers: gcc, lint, splint, Purify

➢ Grokkers: cflow, cscope, ctags, lxr, opengrok

Underlined, Italicized = Proprietary Tools

Page 5

A Program on Disk -

Executable

Courtsey:

http://www.linuxforums.org/misc/understanding_elf_using_readelf_and_objdump.html

Executable and Linkable

Format

Relocatable files (gcc -c)

Shared objs (gcc -shared)

Executable files (ld)

readelf – Read elf headers,

sections and symbol tables

Objdump – Disassemble elf

objects and hack around

Neat Tools: nm, strings, od

Page 6

A Program in Memory - Process

Text Segment – Machine Code.

Shared, Read-Only.

Data Segment – Initialized global

variables from executable

BSS – Has uninitialized global

variables set to zero (Block

started by symbol)

Stack – Collection of stack

frames. Grows Downward

Heap – Dynamic memory for

programs and libraries. Grows

Upward

Low Address

High Address

Page 7

The GNU Debugger (GDB)

➢ Source/Instruction => Process/Core/Kernel(kgdb)

➢ Frequently used commands (TAB Autocompletes)

➢ file/attach – load a binary file for execution

➢ kill/run – the loaded file or process

➢ list – list a file, funcs, lines, addr.

➢ break/clear – breakpoint at func, lines, addr.

➢ step/stepi – single step one source line/mi

➢ next/nexti – step over subroutines

➢ cont – continue until next breakpoint or end

➢ disable/enable – breakpoint manipulation

Page 8

The GNU Debugger (GDB)

➢ Frequently used commands (continued ....)

➢ bt – Print backtrace of all stack frames

➢ frame N – switch stack frame context

➢ display/print – variables, expressions

➢ ptype – print type of variable (stabs/ctf)

➢ info – shows a huge number of useful things

➢ Tools useful in conjunction with GDB

➢ pmap – Display process address layout

➢ elfsh – interactive shell for elfdump !

➢ biew – ncurses gui to explore elf objects

Page 9

Preparing for a GDB session

➢ Don't strip – Cost of Disk << Cost of Engineering

➢ Never ever omit the frame pointer (-fomit-frame-

pointer is evil)

➢ Add the enhanced symbol table (gcc -g)

➢ Disable optimizations when creating a debug

executable (-O0)

➢ Add GNU specific extensions for a lot of extra

debugging power (-ggdb3)

➢ .gdbinit – put redundant commands into this file

Page 10

DDD – GDB for X

Page 11

strace

➢ Powerful runtime tool to trace syscalls and signals.

➢ Restrict to system calls or classes using -e trace

(!)= syscall|set|process|network|ipc|file|desc

➢ Attach to existing process using -p

➢ Follow children of fork() -f and vfork() -F

➢ Coarse profiling using -tt, -r , -c

➢ -s display upto only n characters

➢ Symbol name demangling using -C

➢ Instruction Pointer at the time of trace -i

➢ Log output to a file using -o and -ff

Page 12

ltrace

➢ Runtime tool to trace library calls

(How are library calls and syscalls different)

➢ Aggregate syscalls by count -c

➢ Use ldd to find out static link-time library

dependencies and -l , -L to filter library names.

➢ Indent call flow using -n

➢ Trace system calls to using -S !

➢ Most other options are strace syntax compatible

➢ For ltrace Internals see PTRACE(2)

➢ BUGGY ! ELF32 only ! dlopen() not traced !

Page 13

Profiling Tools

➢ Time from Shell / gettimeofday() / clock()

➢ Instrumentation Profilers

➢ GPROF – Collection/Analysis of execution profile

➢ GCOV - Hotspot detection using code coverage

➢ Quantum Limitations – Heisenberg Principle

➢ Sampling Profilers

➢ oprofile – Kernel, CPU supported counters and

event monitors – understand your CPU well.

➢ AMD CodeAnalyst and Intel vTune

Page 14

Nifty Tools for Unlucky Days

➢ Networking Tools

➢ Wireshark – Brilliant protocol analyzer

➢ Netstat – A lot of useful statistics and views

➢ Netcat – TCP/IP Swiss Army Knife

➢ Nmap – Network Exploration / Port Scanner

➢ Filesystem Tools

➢ fuser – Identify processes using a file/socket

➢ lsof – List of open files. Command line hell

➢ watch (-d)– Repeatedly executes a command.

Waits for output to change. Highlights the change.

Page 15

MM :: Valgrind / Cachegrind

➢ The most advanced MM debugging tool

➢ Use of uninitialized memory

➢ Reading/writing memory after it has been freed

➢ Reading/writing off the end of malloc() areas

➢ Reading/writing to wrong addresses on the stack !

➢ Memory leaks - i.e.malloc() pointers lost forever

➢ Mismatched use of malloc/new[] vs. free/delete[]

➢ Overlapping pointers in memcpy() and friends

➢ Some misuses of the POSIX pthreads API

➢ Memory hog ! 25-75 times slower ! -O0 works best

Page 16

The One True Tool

If you thought Valgrind was mind-

boggling wait until you see what is the

state-of-art in debugging.

DTrace (OpenSolaris, FreeBSD, MacOS)