tool-chain

=======

gdb-scripts

1This repository contains custom GDB commands to enhance debugging capabilities, particularly for heap analysis, process inspection, and deadlock detection. This repository contains:

• Remote Debugging Automation Tool

This tool simplifies remote debugging by automating essential steps required to debug a process on a remote machine. It connects to a remote gdbserver, attaches to a specified process, and facilitates debugging through a local gdb instance. Configuration settings allow you to customize the process according to your system.

---

Table of Contents

• Features
• Requirements
• Installation
• Usage
• Example Command
• Argument Descriptions
• Workflow
• Coredump Debugging
• User Interface (-ui flag)
• Running in Dockerized Mode Wtih out using Wrapper Script
• GDB Custom Commands

---

Features

• Automates setup and connection for remote debugging.
• Installs necessary dependencies for remote debugging.
• Configures gdb paths to recognize downloaded source code.
• Allows auto-loading of shared libraries from the remote machine.
• Enables core-dump debugging by setting GDB pointers correctly.

Requirements

• Python 3.x
• gdb-multiarch
• tmux
• Required Python packages (installed automatically)

---

Installation

1. Clone the repository:

   git clone https://github.com/deep-observability-lab/gdb-script.git
   cd gdb-scripts

2. Install required Python packages:

   sudo pip3 install -r requirements.txt

---

Usage

To initiate remote debugging, run:

sudo ./gdb_script.sh gdb_remote [-h] [-i IP] [-u USERNAME] [-pid PROCESS_ID] [-a ARCHITECTURE]
                           [-ui USER_INTERFACE] [-w WORKSPACE] [-p PORT] [-s SOURCE_CODE] [--docker]

---

## Example Command
```bash
sudo ./gdb_scripts.sh gdb_remote -i 192.168.183.132 -u user -pid 3789 -w ../workspace -p 1234 -a powerpc:common -ui vscode

---

Argument Descriptions

• -h / --help: Show help message and exit.
• -i / --ip: IP address of the remote target (e.g., -i 192.168.5.5).
• -u / --username: Username for authentication on the remote target (e.g., -u root).
• -pid / --process_id: Process ID of the target process (e.g., -pid 1234).
• -a / --architecture: Architecture for the cross-compiled binary. Defaults to auto if not specified (e.g., -a x86_64).
• -w / --workspace: Directory where you should put all the shared binaries/app binaries and source codes.
• -p / --port: Port to set the DEFAULT_PORT. If not specified, GDB uses port 1234 (e.g., -p 1234).
• -ui / --user_interface: Specifies the user interface for debugging. Options: "vscode" for Visual Studio Code or "gdb" for GDB CLI.
• -s / --source: Directory where you should put all the source codes (separated from binaries and libs).
• --docker / --docker: Runs the command inside a Docker container. If specified, the script will execute with Docker-specific configurations.

---

Workflow

1. Run Initial Setup: The script takes the following command as input:

   sudo python3 gdb_remote.py -i 192.168.183.132 -u zahra -pid 3789 -w ../workspace -p 3434 -a powerpc:common

2. Prepare Workspace: In the path you provide script as --workspace, put all the shared binaries, source codes, and executable binary of the process you are going to debug.

3. Install Dependencies: Dependencies will be installed automatically, such as:

   • gdb-multiarch
   • tmux
   • Relevant Python packages

4. Start gdbserver on Remote Machine: After setup, the script initiates gdbserver on the remote machine at the port specified in the input args.

5. Launch gdb Locally: On the local machine, gdb is launched to begin debugging. A custom script is executed within gdb to set the correct paths for the source code.

6. Generate launch.json for VSCode: If you chose vscode for UI mode, you just need to open VSCode at the workspace.

7. Debugging Process: After configuring, the user can start debugging the specified process.

8. Exit and Cleanup: After debugging, the user can run the command:

   exit_gdb

   This command stops the gdbserver on the remote machine and detaches from the process. The local gdb instance will also exit.

---

Coredump Debugging

To perform debugging with a coredump, you can run the following command:

sudo ./gdb_scripts.sh gdb_coredump -w /workspace -a powerpc:common -c core_dump -b program.debug -s /source/code/path/ -ui vscode

Argument Descriptions for Coredump Debugging:

• -w / --workspace: Directory where you should put all the shared binaries, app binaries, and source codes. This should also contain the coredump file.
• -a / --architecture: Architecture for the cross-compiled binary. Defaults to auto if not specified (e.g., -a powerpc:common).
• -c / --coredump: Name of the coredump file located in the workspace.
• -b / --binary: name of the binary program you wish to debug.
• -s / --source: Directory where the source codes are located (preferably in the same directory as the binaries and coredump).
• -ui / --user_interface: Specifies the user interface for debugging. Options: "vscode" for Visual Studio Code or "gdb" for GDB CLI.
• --docker / --docker: Runs the command inside a Docker container. If specified, the script will execute with Docker-specific configurations.

Notes:

• The launch.json for debugging will be automatically generated and placed inside the .vscode folder within your workspace.
• It is recommended to have your source code located within the same workspace directory as the binaries and coredump for better organization.

---

User Interface (-ui flag)

The -ui flag specifies the user interface for debugging, which can either be gdb (command-line interface) or vscode (Visual Studio Code).

-ui gdb:

• If you specify -ui gdb, the tool will automatically open a gdb-multiarch instance in a tmux session on your local machine. This provides an interactive environment where you can run all your debugging commands directly within the terminal.

-ui vscode:

• If you specify -ui vscode, the tool will generate a launch.json configuration file and place it in the .vscode folder inside your workspace. This configuration file will allow you to debug your remote process directly from Visual Studio Code.

Both options are supported for both remote debugging and coredump debugging. Simply choose the appropriate user interface based on your preferences.

---

Running in Dockerized Mode Wtih out using Wrapper Script

You can run both remote debugging and coredump debugging in a Dockerized environment. To do so, follow these steps:

Step 1: Build the Docker Image

Build the Docker image by providing the path to your local repository for gdb-scripts:

docker build --build-arg HOST_PATH=/var/gdb-scripts -t gdb-scripts .

This will create a Docker image with the necessary dependencies. If you want to use the provided image in Dockerhub at docker pull 33zahra/gdb-scripts:latest, clone the repository into the /var/ directory. Then, run the following commands to pull and tag the image:

docker pull  33zahra/gdb-scripts:latest
docker tag 33zahra/gdb-scripts:latest gdb-scripts:latest
docker run -v /path/to/workspace:/work -v /path/to/src:/src -it gdb-scripts gdb_coredump -c core_dump -b myprogram.debug -w /path/to/workspace -s /path/to/src

Step 2: Run Docker Container for Coredump Debugging

For coredump debugging, run the following Docker command:

sudo ./gdb_script.sh gdb_coredump -c core_dump -p myprogram.debug -w /path/to/main-rootfs -s /path/to/src -ui vscode --docker

Important Notes:

• /src: The source code directory should be mounted as a volume to /src inside the Docker container. This is important to ensure that the debugging environment has access to the correct source code.
• /work: The main root filesystem (which includes the binaries and the coredump) should be mounted as a volume to /work. This path should not be changed as it is crucial for proper execution.
• -s: Specify the source code path using the -s argument, pointing to /src.
• -w: Specify the workspace using the -w argument, which should point to /work.

Step 3: Run Docker Container for Remote Debugging

For remote debugging, run

the following Docker command:

sudo ./gdb_scripts.sh gdb_remote -i 192.168.183.132 -u gdb_user -pid 3789 -w ../workspace -p 3434 -a powerpc:common -s /path/to/src -ui vscode --docker -ui vscode

GDB Custom Commands

Command Overview

1. process

Displays general information about the process, including the executable path and PID. (Not available in core dump debugging.)

2. arena

Extracts all heap arenas in the process memory.

3. heap

Displays usage details of all heap memory in the program space.

4. bins

Walks through all bins and checks for corruption.

5. search_pattern <pattern>

Searches for a specific memory pattern within mapped memory ranges. Example usage:

search_pattern 0xb5e00050

this command would search all the open files memory ranges for pattern. if you want to explicitly define ranges, use below format:

search_pattern 0x1006c000 0x1006c0f0 0xb5e00050

this will search for the pattern in the specified range.

6. deadlock

Detects potential deadlocks among threads in the current process.

7. exit_gdb

Custom command to exit GDB.

8. add_child_dirs <parent_path>

Adds all child directories inside a given parent directory to GDB's source path. Example usage:

add_child_dirs /home/victim/src

9.memsummary

Custom command to check summary of heap information.

Usage Order

To ensure proper functionality, use the commands in the following order:

1. process
2. arena
3. heap
4. bins
5. memsummary
6. (Optional) search_pattern if needed.

Installation

Place the command scripts in your GDB initialization directory and source them within GDB:

source path/to/commands.py

Important Notes:

• /src: The source code directory should be mounted as a volume to /src inside the Docker container. This ensures that the source code is available for debugging.
• /work: The main root filesystem should be mounted to /work inside the container. This is where the binaries, libraries, and necessary files for debugging reside.
• -s: Use the -s argument to specify the path to the source code (e.g., /path/to/src).
• -w: The workspace should be mounted to /work in the container.

This setup allows you to run both remote debugging and coredump debugging directly from Docker, keeping all your debugging tools and environment isolated within the container.