ProbeVue

ProbeVue is IBM's implementation of a lightweight dynamic tracing environment introduced in AIX version 6.1.^[1] ProbeVue provides the ability to probe running processes in order to provide statistical analysis as well as retrieve data from the probed process. The dynamic nature of ProbeVue allows it to be used as a global system performance tool while retaining the ability to drill into very specific events on a single process or thread.

Because modifications are not required of a probed process^[2] or system and the lightweight design of ProbeVue as a tracing tool, it is suitable for use in a production environment where previous tracing tools would have been performance prohibitive.

ProbeVue provides a series of probe point specifications that are potential events that can be probed. A script written in the Vue language allows the user to define a probe that is a block of code called an action block that will run when those events occur on the system. The execution of the action block can be limited to specific events by use of a conditional statement placed on the probe called a predicate. The code in the action block follows a C-like syntax with a limited set of built in functions.

The following is an example of a probe that is defined for whenever a process with a PID of 123456 enters the read() system call. When that event happens this script will call the built-in printf() function to print a message to its output trace buffers. The first line in the action block is a C-style comment and therefore will not execute in the ProbeVue environment.

@@syscall:*:read:entry  when ( __pid == 123456) {    /* This is a comment: The process with a PID of 123456 has called read(); */    printf("read() system call entered.\n"); } 

Probes like the above sample can be written and run without the extensive testing normally required of a production system. The ProbeVue environment protects the user from errant code or resource hungry tracing frequently seen with previous IBM tracing tools. The runtime compile feature of ProbeVue provides a powerful ad hoc environment for data gathering.

Currently IBM provides the following probe providers on AIX 6.1: syscall, uft, interval, trace, and the ProbeVue probes (BEGIN and END). The syscall provider defines probe points for a subset of the available system calls. User Function Tracing (uft) probes can be defined for entry points of functions defined in a C or C++ program. Both syscall and uft probes must include a valid function prototype in the Vue script before function parameters (for the entry probes) or return values (for syscall exit probes only) can be accessed. The interval probes fire on a timer and can be defined on 100 millisecond intervals. The trace provider allows ProbeVue to access traditional trace hooks. Finally the ProbeVue probes called BEGIN and END will fire as the probevue environment itself starts and exits.

Because ProbeVue is designed as a tool to monitor operating system events, the Vue language uses an event driven style that is used to describe how to process data derived from these events.

Like most dynamic tracing languages found on other Unices, the Vue language has a C-like syntax and is frequently presented in this context. Vue's relationship to C has many parallels but deviates most significantly from the imperative nature of C. Some of the contrasts and similarities are shown here.^[3]

Because ProbeVue is used to monitor applications written primarily in C, it supports all C data types as well as C data structures. Vue also supports a String, list, and time stamp data types. The String and list types are both actually arrays, while the time stamp is a high resolution representation of a point in time. Type casting and automatic type promotion during operations with mixed types is similar to C behavior as well.

Pointers to arrays and data structures behave in ProbeVue the same as they would in a C program, with the key difference from C is that when pointers refer to a location in memory they are referencing a location in another process space. To access that memory it is necessary to first copy the memory location into the local ProbeVue environment. If the memory has been paged out, ProbeVue cannot force a page fault to access it. Another difference is that the size of pointers in C are determined at compile time, while in ProbeVue they are determined by the 32 or 64 bitness of the application that is probed.

Floating point data types are valid data types for assignment from a probed process, but cannot be used for floating point mathematical operations.

Vue uses scoping rules similar to C, but must also account for the externally derived nature of much of the data in probes. As a result, not all data classes are available or relevant in all probes or all portions of a View script. The basic classifications of data are as follows.

globals - Variables that have scope across the entire Vue script

shell - Variables that follow shell conventions and are read from the Unix environment

kernel - Variables that are provided by the kernel

local - Variables that are local to a probe action block

thread local - Variables that are local to a thread, but have scope across multiple probe action blocks

entry/exit - Variables that are defined by, and local to, the probe

built in - Pre defined variables that have values relevant to the current firing probe

Vue operators follow C conventions closely with the exception of when used with strings. When used with strings, the + operator performs concatenation, and the == operator is used for comparisons.

Vue does not allow the definition of user functions,^[4] recursion, or looping constructs but does offer conditional if-then expressions within a probe action block. The lightweight nature of ProbeVue prohibits a Vue script from defining expensive looping or extensive branching operations that could degrade performance.

Tentative tracing is a concept that allows the trace output of a block of code to be labeled as tentative. The output of this code can later be committed to the trace buffers as visible output or it can be discarded. This works conceptually much like transaction controls in SQL.

A Vue script can be invoked with interpreter magic and set executable like a shell script or can be run as input to the ProbeVue command in the form of a command line parameter or stdin input.^[5] By convention, Vue scripts have a .e filename extension.

The Vue language lacks aggregations and instead uses a list data type that offers similar yet limited functionality. The equivalent product from Solaris called DTrace offers an aggregation data type and a powerful set of aggregating functions to represent statistical data. The list data type offers only basic aggregating functions on a single item (as opposed to an associative arrays of aggregations that DTrace offers). The list data type offers a slight simplification over keeping the stats manually but does not allow the list to be reset (say, over an interval) that can easily be done when using manual summaries and basic types. IBM has committed to associative arrays on future versions of the Vue language.

Because of the long development lead time DTrace has over ProbeVue, DTrace has considerably more probe point specifications.^[6] IBM has plans to add a considerable number of new probe points in future technology level releases of AIX 6.1 and in the next major AIX release.

• DTrace
• SystemTap

• ProbeVue: Extended Users Guide Specification
• ProbeVue QuickSheet
• AIX 6.1 Differences RedBook
• DeveloperWorks article on ProbeVue
• ProbeVue sample scripts and usage notes
• Debugging C++ applications using ProbeVue