15. Datagrind: a data accessor logger

Table of Contents

15.1. Overview
15.2. Using Datagrind and dg_view
15.2.1. Running Datagrind
15.2.2. Running dg_view
15.3. Client requests
15.4. Datagrind output format
15.4.1. Output format structure
15.4.2. Header
15.4.3. Range requests
15.4.4. Event requests
15.4.5. Basic blocks

To use this tool, you must specify --tool=exp-datagrind on the Valgrind command line.

15.1. Overview

Datagrind captures all read and write accesses made by a program and records them in a log file. A separate tool (dg_view), can then be used to visually represent the reads and write.

15.2. Using Datagrind and dg_view

Datagrind does not directly make use of debugging information from the executable, but dg_view uses it to map addresses back to functions and line numbers. Compiling with -g is thus recommended. If you are tuning your program, you should compile with optimisation, as this will show how your program will really access memory. If you are trying to understand your program, it may be more useful to disable optimization so that memory accesses will be done in the same way they appear in source code. Some optimisations (such as inlining) will also make stack traces less accurate.

Once you've built your program, you need to run Datagrind to gather the log of memory accesses. Since every memory access is recorded, this file can get big extremely quickly, and so you should not try to capture more than a second or two of execution (even if you have the storage for a bigger capture, dg_view may struggle to display it).

Once you've captured a log, you can then use dg_view to examine it visually.

15.2.1. Running Datagrind

To run Datagrind on a program prog, run:

valgrind --tool=exp-datagrind prog

The program will execute. Datagrind does not normally output any diagnostic messages. By default, the output log will be written to datagrind.out.pid, but this can be changed by passing the option --datagrind-out-file=filename.out to Valgrind.

15.2.2. Running dg_view

dg_view is a viewer for the output of datagrind, distributed and built separately. It uses GTK+ for display.

To use dg_view, simply run

dg_view datagrind.out.pid

This will open a window showing the data accesses. Green and red dots show read and write accesses respectively; yellows shows pixels where both reads and writes occur. The X axis shows address, while the Y axis shows the instruction count.

To keep the display manageable, not all of the address space is shown. White vertical lines show discontinuities in the address space, where addresses have been omitted. Grey lines show page boundaries, and reddish lines show cache-line boundaries (currently the page and cache-line sizes are hard-coded to 4096 and 64 respectively, rather than obtained from the target machine). The page and cache-line boundaries are only shown when the zoom level is sufficient to avoid them overlapping.

In the initial view, all events and all accessed events are shown, which gives a good overview but makes fine details difficult to see. You can click and drag a rectangle (using the left mouse button) to zoom in on it. The toolbar icons can also be used to zoom in or out in either dimension.

15.2.2.1. Command-line options

To make it easier to focus on specific information, dg_view allows the accesses to be filtered in two ways: by the accessing instruction, and by the memory allocation to which the source or target data belongs. These are controlled by the --events and --ranges options respectively. They each take a comma-separated list of conditions. An access is considered to match if the it matches at least one of the event conditions and at least one of the range conditions. The valid conditions are listed below. Where a reference is made to the call stack, it is the call stack at which the access occurred, for --events, or the call stack at which the memory was allocated, for --ranges. At present, only heap allocations are tracked; if --ranges is specified at all, no accesses to the stack or to global variables will be displayed.

fn:function

Matches if function is in the call stack.

file:file

Matches if a function in the call stack comes from file. This depends on the necessary debug information being stored in the binaries.

dso:dso

Matches if a function in the call stack comes from the executable or shared library dso.

user:string

Matches based on client requests issued by the guest program. Refer to Client requests for more information about client requests.

range

Matches any allocation reported by a DATAGRIND_TRACK_RANGE clent request. Refer to Client requests for more information about client requests. This option is only accepted by --ranges, not --events.

malloc

Matches any access to memory allocated with a malloc-like function. This option is only accepted by --ranges, not --events.

15.3. Client requests

Viewing the whole of space and time (well, within one process) is daunting, and can easily overload dg_view. Furthermore, it does not readily map back to meaningful variables or functions in the source code. To assist with these problems, a number of client requests are made available through the header file datagrind.h.

To reduce the view of space, one can mark particular ranges of memory that are of interest, using DATAGRIND_TRACK_RANGE. Apart from the range itself, this macro takes a string for the type of the memory (currently unused, but may eventually be used to extract DWARF debug information to show structure offsets), and a label, which is a descriptive string. When the memory is freed, or simply ceases to be interesting, the tracking information can be released using DATAGRIND_UNTRACK_RANGE. It is not required to do this (i.e., it will not cause Datagrind or dg_view to crash), but you may start tracking unrelated memory if the memory allocator reassigns the memory elsewhere.

To reduce the view of time, one can mark the start and end of various events, using DATAGRIND_START_EVENT and DATAGRIND_END_EVENT. These macros simply take a label.

This information is simply recorded to the output file. When you run dg_view, you can use the options --events=user:event and/or --ranges=user:range to limit the display to memory accesses that fall within these ranges and events.

15.4. Datagrind output format

For efficiency, Datagrind uses a binary output format to record information. This file format is still in active development and subject to change without notice, and there is no guarantee that old output files will continue to be readable.

15.4.1. Output format structure

The output format consists of a sequence of records. Each record contains enough information for the record to be skipped without understanding it. Values are encoded in the target's endianness; the file contains sufficient information to determine the endianness, although dg_view currently does not support viewing a file with a different endianness to the host on which it is running. Similarly, all addresses and many sizes are stored using the word (pointer) size of the target machine, with sufficient header information to determine this. dg_view currently supports viewing files with either 32-bit or 64-bit words, regardless of the word size of the machine on which it is run.

Each record begins with a byte identifying the record type. For record types 0-127, the second byte is a length field, indicating the length of the rest of the record (excluding the initial two bytes). A length of 255 is special: it indicates that the actual length is encoded in the following 64 bits. For record types 128-255, the record type is followed by a single byte of payload, giving a two-byte record. The numeric values of the record types can be found in dg_record.h.

Strings are null-terminated. The character set and encoding generally depends on the C code that defines the string, but UTF-8 is recommended.

Records are described with a C-like notation, but these structures will not actually be usable for accessing the file, because the file does not contain any padding between fields (thus, many fields are not aligned). The variable-length encoding of the record length (described above) is denoted by a length type.

15.4.2. Header

The first record in the file is a header record. This is the only place the header record may appear. The signature identifies the general structure of the file, and readers that recognise this signature can be confident of parsing the file, with the exception of values that are specifically reserved, and unknown record types.

The header will never use the extended form of the length field, because it cannot be correctly interpreted until the file endianness is known, and the endianness is itself encoded in the header.

struct header
{
    byte record_type;   // DG_R_HEADER
    length record_length;
    char signature[11] = "DATAGRIND1\0";
    byte version;
    byte endian;        // 0 for little-endian, 1 for big-endian
    byte word_size;
};

15.4.3. Range requests

Client requests for range tracking are recorded as records as well.

struct track_range
{
    byte record_type;   // DG_R_TRACK_RANGE
    length record_length;
    word address;
    word length;
    string type_name;
    string label;
};

struct untrack_range
{
    byte record_type;   // DG_R_UNTRACK_RANGE
    length record_length;
    word address;
    word length;
};

15.4.4. Event requests

Like range requests, client requests for event tracking are recorded. The label is truncated to ensure that the record is less than 255 bytes long.

struct event
{
    byte record_type;   // DG_R_START_EVENT or DG_R_END_EVENT
    length record_length;
    string label;
};

15.4.5. Basic blocks

Basic blocks are first defined, then later run. A basic block is actually a superblock, so may cover discontiguous addresses and have internal exits. The block definition describes the read and write accesses in the block, as instruction counts relative to the start of the block. The bbdef_entry fields correspond to IMark and data access tags in the VEX IR.

Basic block definitions are limited to 255 instructions and accesses, and should not cross function boundaries, so they may be smaller than VEX basic blocks.

struct bbdef_instr
{
    word addr;
    byte size;
};

struct bbdef_access
{
    byte dir;            // DG_ACC_READ or DG_ACC_WRITE
    byte size;           // size of data access
    byte iseq;           // index into the instruction array
};

struct bbdef
{
    byte record_type;    // DG_R_BBDEF
    length record_length;
    byte n_instrs;       // number of instructions
    word n_accesses;     // number of data accesses
    bbdef_instr instrs[n_instr];
    bbdef_access accesses[n_accesses];
};

It is common to reach the same basic block with the same execution stack multiple times. Each such event is recorded as a context. It refers to the basic block definition by its position in the file, starting from zero.

struct context
{
    word bbdef_index;
    byte n_stack;
    word stack[n_stack];
};

When a block is run, it will refer to a previously defined context by index (its position in the file). It then contains the addresses of the data accesses, in the same order as the data-access entries in the definition. Because a basic block can be exited early, there is an instruction count in the record, and only the data accesses for this many instructions will be reported.

struct bbrun
{
    byte record_type;     // DG_R_CONTEXT
    length record_length;
    word context_index;   // index into sequence of contexts in the file
    byte n_instrs;        // number of instructions executed before leaving
    word addrs[];         // length determined from record size
};