This article documents the structure of Intermapper chart files.

Note: This information is not intended to allow you to parse a chart file, but rather to give you a better understanding of the underlying data structure.

Pre-6.2 Chart Data File Format

Chart data files contain repeated pairs of eight-byte data:

  • 32-bit UTC time stamp, in seconds since the epoch (1 Jan 1970)
  • 32-bit signed data value (The value 0x80000000 is reserved to represent a discontinuity, meaning that there should be a break between the data values on either side.)

Intermapper appends new data readings to the end of the chart data file in this format. Both these values are in the proper endian order for the current processor.

6.2 format

With the release of Intermapper 6.2, chart data storage has been restructured.

Here are the major differences:

  • Data is now accessible as segments of time rather than file indexes.
  • Time and data values are represented in 64-bit signed or unsigned format.

Why change the format?

The original format had one major advantage. Each data point was a fixed size, with times and data in 32-bit format. This made access to data relatively easy - the number of data points could be calculated by the file size.

It also had some disadvantages:

  • Dynamic purging of data was not an option.
  • Data values could not be greater than 32bits. This presents issues charting high-speed links.
  • The fixed size becomes very inefficient with larger charts.

This new structure permits dynamic purging of chart data.

A new limited retention policy has been added. You can now specify how long chart data will be kept.

No conversion of existing data is required. Old chart data remains untouched, and new data uses the new structures.

As a result of the new data structure, the space required to store chart data has been reduced significantly. Typical storage space reduction is around 40%.

Basic file structure

A new chart data file contains:

  • File Header
  • Data Segments
  • Segment Index

 

File header

  • Signature/version - 8-bytes - Provides a mechanism identify the chart version and to handle version-based structural changes.
  • File start time - Absolute start time for the file
  • File end time - Absolute end time - Absolute start time
  • Data Segment Count- (as defined below) The maximum number of segments is 178,956,970.
  • Minimum data value
  • Maximum data value
  • Segment start offset - specifies the start of the segment index (see below)

Segment structure

  • Segment header
  • Segment data
    • Array of scaled points: points that have strict time variation within [-127, 127]
    • Array of odd points: points that are plainly laid out with [relative-time, value]. The relative-time is calculated by actual time minus the segment absolute start time. The value has a byte size of 1 to 8.

The maximum number of points a segment contains is 251.

Segment header

  • Discontinuity (indicates whether the segment started with a discontinuity point)
  • Polling Interval
  • Checksum
  • Scaled Point Array Data Value Size (number of bytes used to store the data value. Can be between 1 and 8)
  • Scaled Points Count
  • Segment absolute start time
  • Odd Point Array Data Value Size (number of bytes used to store the data value. Can be between 1 and 8)
  • Odd Point Relative Time Size (number of bytes used to store the time)
  • Odd Point Count

Segment body

An array of data entries containing the following:

  • Array of scaled points
  • Array of odd points (if any)

Segment data

To minimize the space required for a data point, Intermapper charts use a default format called a "scaled point".

Scaled points consist of:

  • A signed one-byte Delta Time (-127 to 127) related to the expected time of the data (as calculated using the Scaled Point Count and the polling interval).
  • A signed or unsigned Data Value, with a size specified in Scaled Point Array Data Value Size.

Odd points consist of:

  • Time relative to the start of the segment
  • A signed or unsigned Data Value, with a size specified in Odd Point Array Data Value Size.

For scaled points, the delta time represents the variance from the polling interval. When the data is received the time is subtracted from the expected time to calculate the variance.

Odd points are used only when the variance from the expected time exceeds +/-127.

The storage size of the data value for both scaled points and odd points depends on the type of data being stored. For example, a percentage requires only 1 byte. Storage size can be up to 8 bytes per data point.

Segment Index

The segment index is an array of fixed-size index entries, and it is always in memory when a file is loaded. It provides random access to specific segments without reading from the beginning of the file.

Each entry contains the following information:

  • Segment start time (unsigned 32-bit) relative to file start time
  • Segment end time (unsigned 32-bit) relative to file start time
  • Segment Data Point Count (unsigned 16 -bit)
  • Discontinuity count (unsigned 16-bit)
  • Segment total data size (segment header + actual arrays of data) (unsigned 32-bit)
  • Segment maximum data value (64-bit)

Chart Data Mechanics

When is new file created?

  • When a new chart is added or a new dataset is added to a chart. (Each file contains a dataset.)
  • When the number of segments reaches 4294967295
  • When the difference of absolute ending time and absolute start time is exceeding 4294967295 seconds (which is about 135 years)

When is a new segment created?

A segment is created when:

 

  • When user changes the polling interval
  • When the number of data points in the segment reaches 251.

Finding the Chart Data File which Corresponds to a Specific Dataset

 

The Chart Data folder contains a number of subfolders whose names are “graph IDs”. A graph ID is a unique ID identifying a map, and it takes the form gXXXXXXXX, where XXXXXXXX is a hexadecimal number. Within the subfolder named for a specific map are the individual chart data files. Chart data files are named RYY-CCCC or LYY- CCCC, where YY is a hexadecimal number representing the device’s (R) or link’s (L) id within the map, and CCCC is the four-character code for the type of dataset in question. For example, L2-Util represents the “utilization” statistic for the second link on the map. (N.B. On little-endian machines, such as most Windows and Linux machines, this file will be named L2-litU.)

 

There are several steps for determining the graph ID for a given map is, and within that map, determining the DeviceID or LinkID, and then the four-character code for the particular dataset.

 

The Intermapper GUI’s “export” function exports information about devices and/or links. The “Id” field contains both the map’s GraphID (gxxxxxxxx) and the DeviceID (Rxx).

 

To export devices:

 

  1. In Intermapper or Intermapper Remote Access, open the map you’re interested in.
  2. Choose File->Export->Data File…
  3. In the export dialog that appears, for “Table”, choose “Devices”.
  4. For “Fields”, click on Id, Name, Address, Probe Type and any other information you feel may help you identify individual routers.
  5. Click “Add” to move them to the “Field Export Order” box.
  6. Grab the fields in the “Field Export Order” box and move them up or down to change the order.
  7. Once they are in the right order, click “Export”.
  8. Enter a useful filename and click “Save”.

 

If you look at the resulting file in Excel or a similar tool, you will see name-id pairs such as:

RT.dartware.com. g4c855b97-r74

 

This means that the device named “RT.dartware.com” is on the map with the graph ID “g4c855b97” and has an ID of “r74” within that map. Thus, you will find datasets pertaining to this device in the Chart Data folder named “g4c855b97”, and they will start with “R74”.

 

To export links:

 

  1. In Intermapper or Intermapper Remote Access, open the map you’re interested in.
  2. Choose File->Export->Data File…
  3. In the export dialog that appears, for “Table”, choose “Interfaces”.
  4. For “Fields”, click on DeviceId, InterfaceID, Index, Description, and any other information you feel may help you identify individual links.
  5. Click “Add” to move them to the “Field Export Order” box.
  6. Grab the fields in the “Field Export Order” box and move them up or down to change the order.
  7. Once they are in the right order, click “Export”.
  8. Enter a useful filename and click “Save”.

 

If you look at the resulting file in Excel or a similar tool, you will see something like:

g4c855b97-r106 g4c855b97-L49 10 Intel(R) PRO/1000 MT Network Connection�

This means that the link named “Intel(R) PRO/1000 MT Network Connection”, at index 10 on the device with the ID of r106 on the map represented by the graph ID “g4c855b97” has an ID of “49” within that map. Thus, you will find datasets pertaining to this link in the Chart Data folder named “g4c855b97”, and they will start with “L49”.

Note, too, that even if you’re interested only in link datasets, it will be helpful to have the device export, too, in order to match device IDs corresponding to the device were the link is with device names.

Identifying a dataset

The final step is to find the four-char code for a given dataset type. The table below lists the possible four-char codes. Once you’ve determined the code from the list, you can put it together with the information above to complete the filename.

Device Code List

Big-endian

Little-endian

Notes

kRptrChartPctCollisions

RCol

loCR

 

kRptrChartPctError

RErr

rrER

 

kRptrChartFramePerSec

RFPs

sPFR

 

kRptrChartBytesPerSec

RBPs

sPBR

 

kRptrChartErrPerMin

REPm

mPER

 

kRouterChartPctLoss

PkLs

sLkP

 

kRouterChartRecentPctLoss

RPkL

LkPR

 

kRouterChartAvail

Aval

lavA

 

kRouterChartUPSCapacity

UpsC

CspU

 

kRouterChartUPSTemperature

UpsT

TspU

 

kRouterChartUPSTempFahr

UpsF

FspU

 

kRouterChartUPSLoad

UpsL

LspU

 

kRouterChartUPSAmTemp

UpMT

TMpU

// for environmental probe 1

kRouterChartUPSAmTempFahr

UpMF

FMpU

// for environmental probe 1

kRouterChartUPSRelHumid

UpMH

HMpU

// for environmental probe 1

kRouterChartUPSAmTemp2

Up2T

T2pU

// for environmental probe 2

kRouterChartUPSAmTempFahr2

Up2F

F2pU

// for environmental probe 2

kRouterChartUPSRelHumid2

Up2H

H2pU

// for environmental probe 2

kRouterChartBlitzUsers

BzUs

sUzB

 

kRouterChartAirportUsers

AiUs

sUiA

 

kRouterChartNTPStratum

NtpS

SptN

 

kRouterChartCanopySMReReg

RReg

geRR

 

kRouterChartCanopySMReg

SMRg

gRMS

 

kRouterChartCanopySMSes

Sess

sseS

 

kRouterChartCanopySMLastRSSI

LRSI

ISRL

 

kRouterChartCanopySMAvgRSSI

ARSI

ISRA

 

kRouterChartCanopySMCurrentRSSI

CRSI

ISRC

 

kRouterChartCanopySMLastPower

LPwr

rwPL

 

kRouterChartCanopySMAvgPower

APwr

rwPA

 

kRouterChartCanopySMCurrentPower

CPwr

rwPC

 

kRouterChartCanopySMLastJitter

LJit

tiJL

 

kRouterChartCanopySMAvgJitter

AJit

tiJA

 

kRouterChartCanopySMCurrentJitter

CJit

tiJC

 

kRouterChartCanopySMRTDelay

RTDl

lDTR

 

kRouterChartCanopySMBoxTemp

BTmp

pmTB

 

kRouterChartNexusRSSI

NxRS

SRxN

 

kRouterChartPingRTT

PgRt

tRgP

// (if not down)

kRouterChartPingRTTMin

PgRm

mRgP

// (if not down)

kRouterChartPingRTTMax

PgRx

xRgP

// (if not down)

kRouterChartPingDupCount

PgDc

cDgP

// (if not down)

kRouterChartBuiltinRTT

BiRt

tRiB

// (if not down)

kRouterChartHRMibNumUsers

HrNU

UNrH

 

kRouterChartHRMibNumProcs

HrNP

PNrH

 

kRouterChartUCDCpuUtil

UcuR

RucU

 

kRouterChartUCDMemSwap

UmsP

PsmU

 

kRouterChartUCDMemReal

UmrP

PrmU

 

kRouterChartUCDMemShared

Umsh

hsmU

 

kRouterChartUCDMemBuffer

Umbu

ubmU

 

kRouterChartUCDMemCached

Umca

acmU

 

kRouterChartHRProcLoad

l000

000l

// All processor loads begin with l; array index is rest

kRouterChartHRPctFull

f000

000f

// All storage table % fulls begin with f; array index is rest

kRouterChartUCDLoadAvg

a000

000a

// All load averages begin with a; array index is rest

kRouterChartCustomSNMP

c000

000c

// All custom SNMP stats begin with c...

kRouterChartChartableSNMP

d000

000d

// All SNMP chartable stats begin with d...

kRouterChartCustomTCP

s000

000s

// All custom TCP stats begin with s...

kRouterChartDebugData

DBUG

GUBD

// Auto-generated Debug data...

// In the following entries, X represents a Hex digit.

 

 

kRouterChartAirPortClientRSSI

RIXX

XXIR

// Client RSSI in dBm the last two chars will be filled with // the client id [00..FF]

kRouterChartAirPortClientSignalQuality

SQXX

XXQS

// Client Signal Quality the last two chars will be filled with // the client id [00..FF]

kRouterChartAirPortClientRSSIPct

RPXX

XXPR

// Client in % the last two chars will be filled with the // client id [00..FF]

// Note that we are using everything that xLN xRN xLS xRS xSF xSR xRF xRR xFC xFl xRt

 

kRouterChartProximIfMask

0xFFFFFF00

0x00FFFFFF

 

kRouterChartProximLocalNoise

xLNX

XNLx

// Local noise in dBm last char is the if Idx

kRouterChartProximRemoteNoise

xRNX

XNRx

// Remote noise in dBm

kRouterChartProximLocalSignal

xLS\X

XSLx

// Local signal in dBm

kRouterChartProximRemoteSignal

xRSX

XSRx

// Remote signal in dBm

kRouterChartProximSendFailures

xSFX

XFSx

// Send failures (%)

kRouterChartProximSendRetries

xSRX

XRSx

// Send retries (%)

kRouterChartProximReceiveFailures

xRFX

XFRx

// Receive failures (%)

kRouterChartProximReceiveRetries

xRRX

XRRx

// Receive retries (%)

kRouterChartProximFCSErrorPerSec

xFCX

XCFx

// FCS Errors / sec

kRouterChartProximFailuresPerSec

xFlX

XlFx

// Failures / sec

kRouterChartProximRetriesPerSec

xRtX

XtRx

// Retries / sec

 

 

 

Interface Code List

Big-endian

Little-endian

kLinkChartUtil

Util

litU

kLinkChartUtilTransmit

UtTr

rTtU

kLinkChartUtilReceive

UtRc

cRtU

kLinkChartErrPct

EPct

tcPE

kLinkChartErrPctTransmit

EPTr

rTPE

kLinkChartErrPctReceive

EPRc

cRPE

kLinkChartPktSecTransmit

PktT

TtkP

kLinkChartPktSecReceive

PktR

RtkP

kLinkChartByteSecTransmit

BytT

TtyB

kLinkChartByteSecReceive

BytR

RtyB

kLinkChartBitSecTransmit

BitT

TtiB

kLinkChartBitSecReceive

BitR

RtiB

kLinkChartMegaBitSecTransmit

MBiT

TiBM

kLinkChartMegaBitSecReceive

MBiR

RiBM

kLinkChartMultiPctTransmit

MPTr

rTPM

kLinkChartMultiPctReceive

MPRc

cRPM

kLinkChartBcastPctTransmit

BPTr

rTPB

kLinkChartBcastPctReceive

BPRc

cRPB

kLinkChartErrMinTransmit

EMTr

rTME

kLinkChartErrMinReceive

EMRc

cRME

kLinkChartDiscMinTransmit

DMTr

rTMD

kLinkChartDiscMinReceive

DMRc

cRMD

kLinkChartByteTransmit

CByT

TyBC

kLinkChartByteReceive

CByR

RyBC

kLinkChartPktTotal

CPTt

tTPC

kLinkChartPktTransmit

CPkT

TkPC

kLinkChartPktReceive

CPkR

RkPC

kLinkChartErrTransmit

CErT

TrEC

kLinkChartErrReceive

CErR

RrEC

kLinkChartDiscTransmit

CDsT

TsDC


Still have questions? We can help. Submit a case to Technical Support.

Last Modified On: February 22, 2019