Tracing

This is a tracing library based on the Dapper work by Google. It is also similar to Twitter’s Zipkin and HTrace from Cloudera which were also inspired by the Dapper work.

Tracing is different from simply recording metrics about a service or logging text strings to a log file. Tracing is intended to capture information about the units of work that are being performed, how they relate to one another, and how long the pieces of the units of work take to execute.

Each unit of work at the top level is captured by a trace which consists of a tree of spans (<span>). Each unit of work is tracked by a span in the tree.

Each span can have key/value data associated with it via either trace-add-data or span-add-data. This data typically represents data associated with the computation or action. Examples might include the user name making a request, the SQL being executed, the table being queried, or the IP address associated with a request.

Each span can also have timestamped annotations provided via either trace-annotate or span-annotate. This associates a text description of an event with a timestamp and the span. This might be used to indicate progress through a task, unusual events, or anything interesting.

Other important details are discussed below, such as Writers and Sampling.

Instrumenting Your Code

Instrumenting your code with tracing should be pretty easy.

At application startup, call trace-set-host with a string identifier that is unique among the machines running the software.

To start tracing, either call trace-push and trace-pop or use with-tracing:

with-tracing ("http-request")
  trace-add-data("client-ip-address", ...);
  trace-add-data("requested-path", ...);

  ... do work ...
end;

Setting up Sampling

Sampling of traces is left up to the application. A few basic primitives are supplied in always-sample, never-sample, if-tracing-sample, and related functions. See Sampling for more information.

The TRACING-CORE module

Tracing

The tracing functions in this section represent the high level interface to the tracing library and are what would typically be used, rather than the span-specific functions.

There may be times though when using the lower level, span-specific functions is appropriate, such as when you have multiple units of work executing asynchronously. The asynchronous tasks may find it easier to track their own spans separately.

trace-push Function
Signature

trace-push (description #key sampler) => (span?)

Parameters
Values

  • span? – An instance of false-or(<span>).

Discussion

Create a new <span> and make it the current tracing span. If there is already a span, the new span will use the existing span as the parent.

sampler defaults to if-tracing-sample.

See also

trace-add-data Function
Signature

trace-add-data (key data) => ()

Parameters
Discussion

Adds key / value data to the current trace span (if any), using span-add-data.

See also

trace-annotate Function
Signature

trace-annotate (description) => ()

Parameters

  • description – An instance of <string>.

Discussion

Adds an annotation to the current trace span (if any), using span-annotate.

See also

trace-pop Function
Signature

trace-pop (span?) => ()

Parameters

  • span? – An instance of false-or(<span>).

Discussion

Stops the current span and pops it from the stack, returning the previous span to the current slot.

See also

trace-set-host Function

Sets the global host identifier that will be set on all spans created by this process.

Signature

trace-set-host (host) => ()

Parameters
Discussion

Sets the global host identifier that will be set on all spans created by this process.

This may just be a hostname, but if multiple processes are running on the same host, then it should include additional identifying data. Ideally, this identifier will be unique per process within a cluster in a distributed system.

Note

This function should be called early in the application startup, before any tracing is performed.

See also

with-tracing Macro
Macro Call

with-tracing ("Span description")
  trace-add-data("Table", "users");
  ...
end with-tracing;

with-tracing ("Span description", sampler: never-sample)
  ...
end with-tracing;

Discussion

The with-tracing macro simplifies the process of calling trace-push and trace-pop. It can also take any keywords that trace-push takes and will pass them along.

Spans

<span> Class
Superclasses

<object>

Init-Keywords

  • description –

  • parent-id –

  • trace-id –

Discussion

A span tracks a period of time associated with a computation or action, along with annotations and key / value data. Spans exist within a tree of spans all of which share the same trace-id.

span-add-data Generic function
Signature

span-add-data (span key data) => ()

Parameters
Discussion

Key / value pairs may be stored on a span to provide better context. This might include the query being executed, address or host information or whatever is relevant to the application being traced.

See also

span-annotate Generic function
Signature

span-annotate (span description) => ()

Parameters

  • span – An instance of <span>.

  • description – An instance of <string>.

Discussion

Annotations are to record an occurrence of an event during a span. They have a specific timestamp associated with them that is automatically set to the time when the annotation is created.

See also

span-annotations Generic function

Returns the collection of <span-annotation> associated with this span.

Signature

span-annotations (span) => (annotations)

Parameters

  • span – An instance of <span>.

Values

  • annotations – An instance of <vector>.

See also

span-data Generic function

Returns the property list of data associated with this span.

Signature

span-data (span) => (data)

Parameters

  • span – An instance of <span>.

Values
See also

span-description Generic function

Returns the description of the span.

Signature

span-description (span) => (description)

Parameters

  • span – An instance of <span>.

Values

  • description – An instance of <string>.

span-duration Generic function
Signature

span-duration (span) => (time?)

Parameters

  • span – An instance of <span>.

Values

  • time? – An instance of false-or(<duration>).

Discussion

If the span has not yet been stopped, this returns #f. Once the span has been stopped, the duration that the span was running will be returned.

See also

span-host Generic function

Returns the host identifier for the process which created this span.

Signature

span-host (span) => (host)

Parameters

  • span – An instance of <span>.

Values
span-id Generic function

Returns the unique ID associated with this span.

Signature

span-id (span) => (id)

Parameters

  • span – An instance of <span>.

Values

  • id – An instance of <object>.

span-parent-id Generic function
Signature

span-parent-id (span) => (id)

Parameters

  • span – An instance of <span>.

Values

  • id – An instance of <object>.

span-process-id Generic function
Signature

span-process-id (span) => (process-id)

Parameters

  • span – An instance of <span>.

Values
span-start-time Generic function

Returns the time that the span was created.

Signature

span-start-time (span) => (timestamp)

Parameters

  • span – An instance of <span>.

Values

Like other time values in Dylan, this is the time since January 1, 1900.

See also

span-stop Generic function

Stops a span and sends it to the current registered <span-writer> instances.

Signature

span-stop (span) => ()

Parameters

  • span – An instance of <span>.

See also

span-stopped? Generic function

Has the span been stopped yet?

Signature

span-stopped? (span) => (stopped?)

Parameters

  • span – An instance of <span>.

Values
See also

span-thread-id Generic function
Signature

span-thread-id (span) => (thread-id)

Parameters

  • span – An instance of <span>.

Values
span-trace-id Generic function

Return the trace-id for a span.

Signature

span-trace-id (span) => (id)

Parameters

  • span – An instance of <span>.

Values

  • id – An instance of <object>.

Discussion

Returns the trace-id for a span. This ID is the same for all spans within a single trace.

Annotations

Annotations let you attach events that happened at a point in time (noted by a timestamp) to a span.

<span-annotation> Class
Superclasses

<object>

Init-Keywords

  • description –

  • timestamp –

annotation-description Generic function

Return the description of an annotation.

Signature

annotation-description (annotation) => (description)

Parameters
Values

  • description – An instance of <string>.

annotation-timestamp Generic function

Return the timestamp at which the annotation was created and attached.

Signature

annotation-timestamp (annotation) => (timestamp)

Parameters
Values

Sampling

Samplers allow for collecting a subset of the data, making the usage of this tracing framework in a heavily loaded production scenario more realistic.

Samplers are simply functions that return a boolean value indicating whether or not an actual trace should be generated and recorded. They should be called at the appropriate point within the application being traced. Some applications may wish to limit which traces or parts of traces are collected in ways that are not readily representable within this framework.

always-sample Function

Always returns true, so that the trace is sampled.

Signature

always-sample () => #t

Values

  • record-sample? – Always #t.

if-tracing-sample Function

Returns true if tracing is enabled, otherwise #f.

Signature

if-tracing-sample () => (record-sample?)

Values

  • record-sample? – An instance of <boolean>.

See also

never-sample Function

Always returns false, so that the trace isn’t sampled.

Signature

never-sample () => #f

Values

  • record-sample? – Always #f.

disable-tracing Function
Signature

disable-tracing () => ()

Discussion

This function only modifies the return value of if-tracing-sample and does not globally disable tracing.

See also

enable-tracing Function
Signature

enable-tracing () => ()

Discussion

This function only modifies the return value of if-tracing-sample and does not globally enable tracing.

See also

Writers

Spans are stored by using instances of <span-writer> which have been registered using register-span-writer. Spans are stored when they are stopped (trace-pop, span-stop). Spans are also stored when they are finalized without having been stopped previously. This finalization is only present to prevent data from being lost and should not be a default mode of operation.

<span-writer> Class
Superclasses

<object>

See also

register-span-writer Function
Signature

register-span-writer (span-writer) => ()

Parameters
See also

registered-span-writers Function
Signature

registered-span-writers () => (span-writers)

Values

  • span-writers – An instance of <span-writer-vector>.

See also

store-span Function
Signature

store-span (span) => ()

Parameters

  • span – An instance of <span>.

See also

unregister-span-writer Function
Signature

unregister-span-writer (span-writer) => ()

Parameters
See also

Writer Implementation

To add a new storage class, subclass <span-writer> and implement the span-writer-add-span method. Then, call register-span-writer with an instance of your span writer and all subsequent spans completed will be written to it.

span-writer-add-span Generic function
Signature

span-writer-add-span (span span-writer) => ()

Parameters
Discussion

This method is specialized for each subclass of <span-writer>. It is called whenever a span needs to be processed by a span writer.

Time Utilities

<duration> Class

Measure of time elapsed.

Superclasses

<object>

Init-Keywords

  • microseconds –

  • seconds –

See also

duration-microseconds Generic function
Signature

duration-microseconds (duration) => (microseconds)

Parameters
Values
See also

duration-seconds Generic function
Signature

duration-seconds (duration) => (seconds)

Parameters
Values
See also

<timestamp> Class

A point in time.

Superclasses

<object>

Init-Keywords

  • microseconds –

  • seconds –

See also

timestamp-days Generic function
Signature

timestamp-days (timestamp) => (days)

Parameters
Values
See also

timestamp-microseconds Generic function
Signature

timestamp-microseconds (timestamp) => (microseconds)

Parameters
Values
See also

timestamp-seconds Generic function
Signature

timestamp-seconds (timestamp) => (seconds)

Parameters
Values
See also

Tags

These constants are available to help standardize tracing across applications.

$tag/peer/host-name Constant
$tag/peer/ip Constant
$tag/peer/port Constant
$tag/http/uri Constant
$tag/http/response/size Constant
$tag/http/status Constant

Miscellaneous

get-unique-id Function
Signature

get-unique-id () => (id)

Values

  • id – An instance of <unique-id>.