• Goals
    • Understand the principles of InterMapper Probes
    • Learn to create your own InterMapper SNMP, TCP, and Command-line probes
  • Prerequisites
    • Familiarity with InterMapper
    • Some knowledge of SNMP helpful
    • Some knowledge of your network gear
  • What InterMapper Probes Do...
    • InterMapper Probes are software plugins that perform one or more tests on a particular network device
    • Every time InterMapper "probes" or tests a device, it can perform the following operations:
      • Retrieve specific information from a device
      • Make calculations using the retrieved data, e.g. convert Fahrenheit to Celsius temperatures (optional)
      • Compare the retrieved and calculated values to thresholds
      • Set the device status, severity, and condition string based on those comparisons
      • If configured, a change in severity can trigger a notification
      • Display relevant information in the Status Window so the customer can see the current state of the device
    • Review - Looking at the variety of InterMapper Probes
      • Set Probe... window shows all the probes. A number of them are built-in (part of InterMapper as it ships), you can create more.
      • Each probes has a description, and optionally parameters that customers can set to control the test for a device.
      • Basic category: Automatic switches to either Ping/Echo or SNMP Traffic; Map Status used for building submaps
      • SNMP category: Single OID, Table Viewer, Trap Viewer, High & Low Threshold, High Traffic/PPS/Utilization
      • Network Devices category: Probes for gear from various vendors
      • Servers-Standard category: Probes for various types of "standard" servers. HTTP, DNS, FTP, Printers, Radius, Telnet, etc.
      • Servers-Proprietary category: Probes for vendor-specific servers.
      • Probe Groups category: Another way to create a probe group.
      • WMI category: A set of probes for querying Windows Management Instrumentation (WMI) info from a Windows box. Only available on InterMapper servers running on a Windows computer.
      • Miscellaneous category: A place to put probes that you don't know where they belong.
      • Experimental: A set of probes that are under test. A useful place to put your probes while you're creating them.
      • Wireless category: A set of probes targeted at fixed wireless broadband providers (WISPs)
      • Nagios: Although not a distinct category, Nagios plug-ins provide a wide set of tests that enhance InterMapper's built-in probes. InterMapper can launch plug-ins directly, or can run them remotely using the NRPE protocol.
  • What Probe Designers/Creators Must Do...
    • Determine which information best represents the device's operational health
    • Determine how to retrieve that information (say, via SNMP, TCP, or by running a script or program)
    • Determine how to compare the retrieved data to thresholds (if any)
    • Determine the severity to be assigned to the device when the thresholds are crossed
    • Determine how to display the important information to the customer
    • And finally, create a "probe file" that accomplishes all this
    • Review - Important questions as you start an InterMapper Probe
      • When you start to work on a probe, you need to collect information about the equipment and what you want to measure and alarm on. The questions include:
        • What kind of equipment do you want to monitor?
        • What information do you want to collect?
        • How does the device make it available? SNMP? TCP? Web interface? Other?
        • What thresholds do you want to compare against? Who decides what the thresholds should be?
        • How would you like to see the data displayed? What will be shown in the Status Window?
        • How will you test the resulting probe? Will you have direct access to the equipment during development?
      • Note: None of the questions above require programming or arcane knowledge. You only need cooperation from a person who is familiar with the equipment. They're often best answered on pen and paper before getting into the details of making a probe.
  • A Word About Status, Severity, Condition Strings and Notifications
    • Status shows the device's state. It can be: Up, Down, Unknown
      • Up - The device being tested returns a response to queries (whether the response contains good or bad news)
      • Down - The device does not return any response
      • Unknown - A transitory state: InterMapper hasn't polled the device yet, or isn't currently polling it (set to non-polling)
    • Severity is a subset of the Up status. The severities are: Warning, Alarm, Critical, Minor, Major
      • OK - the icon is green or normal color, the device is operating as expected
      • Warning (also called Minor if you're using Alarm Points) changes the icon color to yellow, lowest threshold, least serious severity
      • Alarm (Major for Alarm Points) gives orange color, higher threshold, medium severity
      • Critical gives red (non-flashing) color, highest threshold, most serious severity
    • Condition String is text that appears as the "Reason" in the Status Window.
      • Set as a result of the probe's comparisons of values to thresholds
      • Probe designers establish these thresholds and create the text of the Condition String when creating probes
    • Notifications (or Alerts) are sent when the device makes a transition to a particular state
      • InterMapper maintains a global list of notifiers that define how an alert can be sent, e.g., email, pager, SMS/text message, sounds, running a script, etc.
      • When configuring a device on a map, you indicate that an alert must be sent using a particular notifier when the device makes a transition to a specific status/severity (up/down/warn/alarm/critical).
    • Review - A Probe that shows all severities
      • The "Status Sequence" probe cycles through a sequence of status/severities. Each time you reprobe, the device gets a new status/severity. Retrieve the probe from http://download.intermapper.com/contrib/probes/com.dartware.snmp.status_sequence.txt
      • Import the probe file into InterMapper (File -> Import -> Probe...)
      • Create a map, add a device (localhost is a good address) and set the probe to Miscellaneous/Test/Status Sequence.
      • Reprobe (Ctl/Cmd-K) the device to see its icon change color
      • Open the Status Window and reprobe to see the condition string change.
  • Types of InterMapper Probes
    • There are several types of InterMapper probes. Each type retrieves information through a different mechanism.
    • SNMP probes retrieve values via SNMP queries or traps from a device
      • Example Probes: SNMP Traffic, High Traffic, Table Viewer, Trap Viewer
    • TCP probes create a TCP connection to a device, requests information
      • Example Probes: HTTP, SMTP, IMAP, many others
    • Command Line probes run a script to get information
      • Example Probes: WMI, POP & IMAP Round trip, Nagios Plugin
    • Big Brother probes listen for Big Brother status reports and update the device status. These cannot be edited.
    • Built-in probes are built into InterMapper's base capabilities. These probes cannot be edited.
      • Example Probes: Ping, DNS, many others
  • Format of an InterMapper Probe File
    • The Developer Guide is the reference for all InterMapper probe development. This syllabus has many links into that manual.
    • Example Probe Files
      • MIB Viewer Probe Builder. A good way to review a well-formatted SNMP probe (and by extension, other probes) is the SNMP "MIB Viewer" Probe Builder web page. This creates a probe that displays all the variables available from a MIB file that you paste into the page. If you click Create the Probe without pasting in a MIB file, you will see an correct probe file, but without any variables. Open this page now, in a separate window while you're reading this document.
      • Another source of probes is to unzip the BuiltinProbes.zip file InterMapper Settings / Probes folder. The files contain the actual text of all the probes that ship with InterMapper. This is a useful way to see how complex probes were designed.
    • InterMapper Probe files use an XML-ish format, however they are not real, proper XML
      • InterMapper Probes are composed of a series of sections, each with different functions
      • Sections are delimited by <section_name> and </section_name> tags. Tag names are not case-sensitive.
    • All InterMapper probes may have the following <sections>. Required sections are marked (Required).
    • <description> section (Optional)
      • Human readable text that describes the probe: its functions, its options, its output, etc. The description text appears in the Set Probe... window. This should contain as much information as is necessary for the customer to understand the purpose and operation of the probe.
      • Text can be styled (bold, italic, colored, linked) using InterMapper Markup Language (IMML) Example:
        <description>
           This is a sample probe description. You should include information about the 
           probe's input parameters, what the probe will do, and the 
           information that will appear in the Status Window. 
           It can have \b\bold\p\ or \i\italic\p\ or other IMML text styling.
        </description>
    • <header> section (Required)
      • Defines several properties of the probe. The property name may optionally be written in quotes (e.g., "type" or "display_name"). The property name is not case-sensitive. Property names include:
        • Human Name (shorthand name, shown in Status Window)
        • Display Name (shows hierarchy in Set Probe... window) Example: "Miscellaneous/Test/Calculation Tester"
        • Probe Type (SNMP, Command-line, TCP, etc.)
        • "Package" (usually a reversed domain name, like "com.dartware") and "Probe_name" (a descriptive name). When combined together with a "." between, these form a unique internal name for the probe.
        • Assorted flags and address types. See the Developer Guide for details.
        • Default port number. Customers can change this when setting other probe parameters in the Set Probe... window.
        • Version. The format is "#.#" (e.g., "2.1") InterMapper will always use the largest version number.
        • URL_hint. This gives a URL to launch if the icon for this device is double-clicked.
        • old_protocol and old_script: These are deprecated and are ignored, but may be found in some probes.
    • <parameter> section (Optional)
      • Customer-settable values for the probe to use as parameters or thresholds. These have the form:
        • "variable_name" = "default_value" where variable_name is the name of the variable, and default_value is a user-settable value.
        • "variable_name" can be referred to as $variable_name or ${variable_name} within the probe. Use the latter format (with "{ ... }") if variable_name has spaces.
        • "variable_name*" is treated as a password; typed characters appear as bullets.
        • "variable_name[Choice1,Choice2,Choice3]" = "Choice3" is a dropdown. The items in the dropdown would be Choice1, Choice2, and Choice3.
        • "variable_name[true,false]" = "false" is shown as a checkbox.
        • Parameter Test probe: [Retrieve the probe file]
    • Display sections (Optional). The contents of the <display...> section control the data shown in the bottom of Status Windows
      • The contents of the display section serves as a template. Variables are filled in with their values. The contents are marked up in IMML.
      • Different names for different probe types
        • <snmp-device-display> for SNMP probes
        • <script-output> for TCP probes
        • <command-display> for Command-line probes
      • Example <snmp-device-display> section from an SNMP probe:
        <snmp-device-display>
        \B5\RFC1213-MIB Viewer\P0\
        \4\ sysDescr:\0\ $sysDescr_ \3G\sysDescr\M0\
           ...
        \b5\Tables from RFC1213-MIB\P0\
           ${ifTable:View the ifTable}
        </snmp-device-display>
      • In the example above, the value of the variable $sysDescr_ was expanded into the template; the ${ifTable:...} macro became a link that if clicked opens the on-demand table. The assorted IMML markup in "\...\" changes the styling, color and font of the text.
    • <Autorecord> section (Optional)
      • When InterMapper reports to InterMapper Database, it can automatically record certain datasets into the database even if they are not being charted. Including entries in an <autorecord> section causes the named variable's values to be sent automatically in the database. For more details, datasets
      • Example:
        <autorecord>
          $var1, 'label1', "Legend 1"
          $var2, 'label2', "Legend 2"
        </autorecord>
    • Other characteristics of probe files:
      • Comments
        • Single line - Everything after "--" on a line is ignored
        • Multi-line HTML-style comments. Example:
          <!--
          This is a multi-line comment
          -->
      • InterMapper Markup Language (IMML)
        • A set of formatting (markup) commands that can be applied to text in Status Window Display sections, probe descriptions, chart legends, condition strings, and other places.
        • Markup commands are placed between matching backslashes, e.g., "\b\" means the following text is bold; "\p\" returns to "plain" text. A complete description is in the Developer Guide
    • Review - Examine the MIB Viewer probe builder page, and the probe that it creates
      • Open the MIB Viewer Probe Builder web page in a separate window.
      • Click Create the Probe to make an empty SNMP probe.
      • Notice each of the sections that are listed:
        • Comments You will see comments both as <!-- etc... --> and comments following "--"
        • <description> A small amount of IMML markup used to make the heading bold text
        • <header> Sets various properties including probe type, package and probe_name, and others
        • <parameters> None for a MIB Viewer probe, but this would be the place to add them
        • <snmp-device-properties> None for a MIB Viewer probe
        • <snmp-device-variables> If you pasted a MIB, the scalar variables will be defined with format_option of "DEFAULT"
        • <snmp-device-variables-ondemand> If you have pasted a MIB, the tables will be defined here.
        • <snmp-device-thresholds> None for a MIB Viewer probe
        • <snmp-device-display> One line per scalar that was defined, one line per on-demand table defined by the MIB file.
  • Digression - A Bit About SNMP
    • Before we talk about SNMP Probes, it's useful to have a bit of background about the SNMP protocol itself.
    • SNMP is an acronym for the Simple Network Management Protocol
    • A management station (such as InterMapper) sends UDP-based SNMP request for one or more values to a device, usually on port 161
    • An agent on that device listens for requests, looks up desired values and sends a (UDP) response back.
    • A trap is an unsolicited delivery of information from the agent on a device. When the device detects an important condition, its agent can send a trap packet to InterMapper with a number of variables ("VarBinds") that describe the event. Traps usually arrive on port 162.
    • An inform is like a SNMP trap that has an acknowledgement. Traps may be lost since they're merely UDP packets sent via best-effort delivery. An agent will resend inform messages until it receives a corresponding acknowledgement. Informs are only available for SNMPv2c and SNMPv3.
    • All values that can be retrieved via SNMP have a name, called its ObjectID or OID
    • OIDs for a particular kind of device are defined in a Management Information Base (MIB), which is nothing more than a text file that specifies the variable's MIB names, the corresponding OIDs, and the definition of the variables.
    • MIB names are much more convenient to type than OIDs. For example, it's easier to deal with the MIB name ifNumber.0 (which represents the number of interfaces on a device) and its OID 1.3.6.1.2.1.2.1.0 even though they are completely equivalent in InterMapper and most other SNMP utilities.
    • A MIB Browser program (such as Mibble - see below) lets you import the MIB file to see what values are available, their exact definitions, and their corresponding MIB Name and OID.
    • Review - Sources of SNMP Information
      • The MIB Viewer Probe Builder page also has links to:
        • Dartware's collection of industry standard MIBs. These come from the libsmi project.
        • Cisco's collection of MIBs for their equipment
        • The MIB Depot collection of MIBs, which have been contributed from people around the Internet. If you cannot get a MIB from your vendor, a search the MIB Depot site may find your equipment.
        • Links to the SimpleWeb MIB validator utility. This tool checks whether a MIB is properly formatted. In addition, it lists several other MIBs that may be useful.
        • Finally, the MIB Viewer Builder page has links to a downloadable version of the page to run locally.
        • Mibble is a SNMP MIB Browser program. It opens MIB files and displays the definitions of all the MIB variables defined there.
  • SNMP Probes
    • SNMP Probes - All SNMP Probes perform these steps:
      • Retrieve data (one or more OIDs) using SNMP GetRequests or GetNextRequests
      • Evaluate the retrieved data against thresholds
      • Set device status/severity based on those thresholds
      • Display collected info in Status Window
      • NB: Arrival of any trap to an SNMP probe will cause an immediate re-poll of the device. This allows the probe to see if there is new information that needs a response. Many customers never need to write SNMP Trap probes because the re-poll facility addresses the need to have a timely response to a fault.
    • SNMP Probe File Syntax
      • Sample Probe: examine the MIB Viewer Page (below) or the built-in Single OID probe
      • Use "custom-snmp" as the type in the <header>, as well as these sections:
      • <snmp-device-variables> section specifies OIDs or MIB names to retrieve. Each variable is defined on its own line, and has the format:
        variable_name, OID_or_MIB_name, Format_Option, "Legend_in_a_chart"
        Each of these is defined below:
        • variable_name is the name of the variable that will be assigned the value from the device. A variable_name consists of letters, digits and an underscore, and must begin with a letter. variable_name's are not case-sensitive.
        • OID_or_MIB_name can either be a SNMP OID or the name of an OID from a MIB that InterMapper has imported. Note: SNMP variable names are case-sensitive, and must exactly match the value specified in the MIB. Note:
          • The OID_or_MIB_name for a scalar (non-table) value should end in ".0" For example, the equivalent values - ifNumber.0 and the OID 1.3.6.1.2.1.2.1.0 both end in ".0"
          • The OID_or_MIB_name for a table value should always end with its row number, e.g., ".row_number" For example, ifInOctets.1 gives the value of the received number of octets on interface number 1.
          • You can use a <parameter> as part of an OID_or_MIB_name. For example, if you have created a <parameter> named "ix", you can enter ifInOctets.$ix in the OID_or_MIB_name. InterMapper will substitute the value of the parameter ($ix) to determine which OID to retrieve.
        • Format_Option controls the format/representation of the value returned.
          • The DEFAULT format option displays values in their most useful format.
          • The "CALCULATION" format_option allows you to set a variable based on the result of a calculation.
          • Read about other Format_Options in the Developer Guide.
        • Legend_in_a_chart (optional) is a string that becomes the legend for the data in a chart file.
      • Each time InterMapper probes a device, it processes the <snmp-device-variables> section in top-to bottom order. You may use the results of an earlier line in the calculations or OID_or_MIB_names. InterMapper:
        • Retrieves the value of OID_or_MIB_name(s) from the device.
        • Sets variable_name to that value
        • Formats that variable according to the Format_Option specified
        • The variable may be used anywhere in the probe with the name $variable_name or ${variable_name}. Use the latter format if the variable_name contains spaces.
      • <snmp-device-variables-ondemand> section (Optional) Specifies the columns of one or more SNMP tables. On-demand tables are used throughout MIB Viewer probes. These variables are not queried unless they are specifically requested by clicking in the Status Window, hence the name "on-demand".
        • The first line specifies the name of the table, and has the form:
          table_name, the_OID, TABLE, "Short description of the table"
          • table_name is an arbitrary name. Recommendation: use the name from the MIB
          • the_OID must be ".1"
          • "Short description..." appears in the header of the on-demand table
        • Subsequent lines specify the columns, and have the form:
          table_name/column_name, the_OID, Format_Option, "Column Heading"
          • table_name must be the same as the first line
          • column_name is an arbitrary name. Recommendation: use the name from the MIB
          • the_OID must be the OID or name of the column as specified in the MIB
          • Format_Option has the same set of choices as <snmp-device-variables>
          • Column Heading is text that appears as the column's heading when the on-demand table appears.
        • To display a table in the Status Window, add ${table_name} or ${table_name:Human Readable Name} to the <snmp-device-display> section
      • <snmp-device-thresholds> section (Optional) Specifies of comparisons to set the device status/severity.
        • The thresholds are a list of lines, with a severity and expression, and optional text for the condition string on each line. For example:
             down: ($var >  7) "Condition string for Down severity"
            alarm: ($var >  5) "Condition string for Alarm severity"
          warning: ($var >  3) "Condition string for Warning severity"
             okay: ($var <= 3) "Condition string for Okay severity"
        • InterMapper always evaluates thresholds from the top to the bottom. In this example it compares the variable $var in each expression. If the expression is true, then the device is set to that severity, the condition string (in the Status Window) is set to the corresponding text, and InterMapper stops evaluating the remainder of the thresholds. (Consequently, you should place the most severe conditions at the top of the list.)
      • <snmp-device-properties> section (Optional) Specifies assorted flags that govern probe operations. You do not usually need to set them, but the two most frequent are:
        • pdutype="get-request" By default, InterMapper sends SNMP get-next-request packets to the device. Certain equipment cannot handle these, and this property forces InterMapper to use SNMP get-requests.
        • maxvars=1 By default, InterMapper places as many OIDs (variables) as possible in a request packet. Certain equipment can handle only one OID request per packet. Setting maxvars to 1 will force a single OID to be sent per SNMP request packet.
      • <snmp-device-alarmpoints> section (Optional) Specifies individual testable and acknowledgeable data values and thresholds. These are deprecated.
    • Troubleshooting SNMP Probes
      • Check connectivity. Make sure that you have the correct device address, the proper SNMP read-only community string, and that the device is listening on port 161. Make sure that port 161 is open in any firewalls on the InterMapper server or that may exist between InterMapper and the device being tested.
      • InterMapper will try SNMP queries twice, then fall back to a ping request. If you see a Status Window with "No SNMPv1 response" or "No SNMPv2c response", it means that the device is not answering InterMapper's SNMP queries, but is answering pings.
      • If the Status Window shows the sysLocation or sysContact, then you know that SNMP information can be retrieved.
      • If a variable is shown as [noSuchName], then this means that the device does not support that particular variable. Check the OID or MIB name as entered in the probe file.
      • If the Status Window shows "tooBig error", then the SNMP response is too large to fit in a single packet. Use the <snmp-device-properties> section to set maxvars to a smaller value. Setting maxvars=1 will guarantee that the response will fit in the packet, at the expense of increasing the number of packets sent through the network.
      • Some devices only respond to GetRequest queries. If no responses return, use the <snmp-device-properties> to set the pdutype="getrequest". This will force InterMapper to use the SNMP GetRequest query, instead of its default GetNextRequest.
      • Use the "SNMP/Single OID" probe to request the desired variable.
      • Use the Monitor -> SNMPWalk command to request the OID, or to walk a number of OIDs from the device.
      • When creating and importing a probe, syntax errors will be displayed in the Event Log when you Reload Probes.
      • Be sure you have imported a MIB file before importing or editing a probe file that relies on the variables defined there.
      • Use a packet sniffer (Wireshark, tcpdump) to see if the traffic is arriving
    • Review - Creating and Editing an SNMP Probe
      • Open the MIB Viewer Probe Builder web page in a separate window.
      • Paste in a MIB. RFC1213-MIB (also known as "MIB-II") is a good choice.
      • Import the probe that you created. To do this:
        • Copy/paste it to a text file
        • Save it using the filename suggested at the top of the file
        • Import using File -> Import -> Probe...
        • NB: You can install the MIB Viewer Builder directly into your InterMapper server and import the probe and the MIB with a single click. The ReadMe file in the downloadable archive gives the steps to do this.
      • Add an SNMP device to a map, and set its probe to "MIB Viewers/RFC1213-MIB Viewer".
      • Open the device's Status Window. Observe that there are many scalar variables.
      • Click the "View the ifTable" link at the bottom of the Status Window. Observe that an on-demand table appears, showing all the rows and columns defined by the ifTable.
      • Edit the probe file, and observe the changes to the Status Window. To do this:
        • Open the probe file with a text editor. The RFC1213-MIB probe will be in:
          • InterMapper Settings/Probes/MIB_Viewers/com.dartware.viewer.rfc1213-mib.txt
        • Find the <snmp-device-variables> section, and remove (or comment-out, using "--") several of the variable definitions, e.g., sysContact.0 and sysLocation.0.
        • Save the file. It will replace the previous version, but will not take effect until you Reload Probes.
        • Reload probes by using Ctl-Shift-R (or Cmd-Shift-R on Macintosh). You will see that the probe was imported. Or choose Monitor -> Set Probe... and click the Reload icon in the lower left corner.
        • Open the device Status Window, and see that your changes have been reflected in the window.
  • Functions, Macros, and Calculations in SNMP Probes
    • InterMapper has many macros and functions that format information within a probe. They are documented in the Developer Guide.
    • All functions and macros have the following format: ${function_or_macro_name: param1, param2, ... } There must not be a space between the function_or_macro_name and the ":". InterMapper evaluates the function (or macro) using the parameters that are passed in, and replaces it with the result of the expression
    • For example, the ${chartable: #.## : $var} macro causes the variable $var to be displayed in the status window as an underlined value that can be charted. The #.## is a formatting option that shows how many decimal places should be displayed. (You may also use sprintf %f5.1 notation in place of the #.##)
    • Calculations for SNMP Variables: The CALCULATION format_option lets you set a variable to the result of a calculation.
    • Built-in Variables and Macros: InterMapper has a broad set of facilities for calculating and displaying information.
    • Persistent Variables: InterMapper will remember variables between probe executions, so that the subsequent probe can compare a result to the prior invocation.
    • Review - Calculations, Macros, and Functions
  • SNMP Trap Probes
    • All SNMP Trap Probes perform these steps:
      • SNMP Trap Probes listen for arriving trap packets.parse the data contained there (the VarBind list items) into variables, just as if InterMapper had explicitly requested the variables.
      • When a trap arrives, parse out the Variable Bindings ("varbinds") from the trap. Assign them to the variables defined in the probe.
      • Compare all the variables received from the trap against thresholds
      • Set device status/severity based on those thresholds
      • Display collected info in Status Window
      • In addition, InterMapper logs the arrival of the trap, along with its VarBinds into the Event Log file.
      • If the MIB file that describes the trap has been imported, InterMapper will use the associated text for the OIDs and values instead of displaying the OIDs and/or numeric values.
      • NB: SNMP Trap probes never poll for any data, unlike the SNMP Probes described above.
      • NB: Whenever a trap arrives from a device, InterMapper always re-polls the device. This frequently detects all important conditions, and many customers find that it's not necessary to write special SNMP trap probes.
    • SNMP Trap Probe File Syntax
      • Sample Probe file: Use the SNMP/Trap Viewer probe, com.dartware.snmp.trapdisplay.txt
      • Use "custom-snmp-trap" as the type in the <header>
      • Other <sections> are the same as an SNMP probe.
      • There is a special TRAPVARIABLE Format_Option in the <snmp-device-variables> section. When a trap packet arrives with this OID, InterMapper assigns the corresponding variable to have the value. If a particular OID is not present in the trap, its variable is set to #ff0000.
      • As with the SNMP Probe, use the <snmp-device-thresholds> to set the device severity based on the values of the received trap variables.
    • Troubleshooting SNMP Trap Probes
      • Make sure that you have the device address correct, and that the device is configured to send traps to InterMapper's address
      • Make sure that port 162 is open in any firewalls on the InterMapper server or that may exist between InterMapper and the device being tested.
      • Use a packet sniffer (Wireshark, tcpdump) to see if the traffic is arriving
    • Review - SNMP Trap Probes
      • Add a device to a map. Set its IP address to a physical device that will send traps.
      • Set the probe to be SNMP/Trap Viewer and notice that the Status Window displays the VarBind fields of the most recent trap that has been received
      • Examine the SNMP/Trap Viewer probe in a text editor. Retrieve it from: http://download.intermapper.com/contrib/probes/com.dartware.snmp.trapdisplay.txt
      • The probe shows how to define trap variables from the MIB, or how to use the positional values (VarBindValue1, VarBindValue2, etc.)
      • The <description> section of the probe has examples of the net-snmp snmptrap command that can send a trap to the probe for testing.
  • Command-Line Probes
    • All Command-line Probes perform these steps:
      • Run a program/script as if you typed the command "from the command line"
      • The command line arguments can pass in thresholds on the command line. The stdin file can also pass information.
      • The script executes, collects data, and evaluates it against any supplied thresholds
      • The script then returns an OS status and optionally use its stdout file to return certain variables and a condition string.
      • InterMapper uses the returned OS status to set the device status/severity
      • Information from the stdout file are formatted and displayed in Status Window
      • NB: The Nagios Plugin probe is a built-in probe that will execute any Nagios plugin that has been saved to the Tools directory.
    • Command Line Script API
      • InterMapper cd's to the path directory before executing the script. This means that the script can save files locally between invocations.
      • InterMapper concatenates path, cmd, and arg to create the command line.
      • InterMapper passes the input as stdin to the resulting script.
      • The script returns two items:
        • $Exit_Code is set to the OS return value from the script. This is used by the <command-exit> section to set the device severity
        • The stdout file contains a set of variables and an optional condition string. The output must have the format:
          \{ $var1 := 5, $var2 := 6 } Condition-String
        • The variables ($var1, $var2) are automatically parsed, and may be used in the <command-display> section
        • The Condition-String is also parsed, and will appear as the Condition string in the Status Window.
        • NB The initial "\" in stdout is important, and cannot be omitted.
        • Whenever InterMapper reloads probes, it tests to see if the probe has a <tool:programname> section. If so, it:
          • Creates a folder in InterMapper Settings/Tools. The folder is named with the probe's package.probe_name (e.g., com.dartware.tool.noaa) If the folder already exists, all non-hidden files within that directory are deleted.
          • Creates a new copy of the programname file into that folder
    • Command-line Probe File Syntax
      • Sample probe - NOAA Temperature Probe
      • Use "cmd-line" as the type in the <header> as well as these sections:
      • <command-line> section (Required) Specifies the script to run and arguments to pass to the script
        • path - the path to the program/script enclosing folder
          • If empty, defaults to InterMapper Settings/Tools or InterMapper Settings/Tools/package.probe_name if there's a <tool:programname> section
        • cmd - the name of the program or script to execute
          • The ${PYTHON} macro invokes the built-in IMDC Python interpreter (with the "-OO" argument) so that Python programs are guaranteed to work cross-platform and without regard to other Python interpreters. (InterMapper 5.5 installs Python 2.7.1)
          • If using ${PYTHON}, leave path = ""
        • arg - arguments to the script. Can include user-entered <parameter>s
        • input - a single line of text enclosed in double-quotes ("...") that is passed as stdin to the script. This is useful for passing credentials (such as username/password) to the script rather than passing them on the command-line, so they will not be visible to a "ps" listing, for example. It is also convenient for passing arguments that may contain whitespace.
      • <command-display> section (Optional) Specifies a template for information to be shown in the Status Window
        • Variables are parsed from the script's stdout and may be displayed in the window.
        • The Reason/Condition String is always shown in the Status Window.
        • The ${nagios_output} macro can be placed in the <command_display> section. This macro parses a Nagios plugin's PERFDATA from stdout and automatically displays the data.
      • <command-exit> section (Required) Compares the exit code of the script and selects proper status/severity.
        • Example:
          <command-exit> 
           down: $Exit_Code = 2 
          alarm: $Exit_Code = 1 
           okay: $Exit_Code = 0 
           </command-exit>
        • $Exit_Code will always be set to the script's OS exit code. In the example above, if the $Exit_Code is 2, the device status will be DOWN; if it's 1, set status/severity to alarm; if it's 0, set it to Okay.
      • <tool:filename> section (Optional) Brackets the source code of a "companion script" within the probe file itself. The script file will be saved in the Tools directory as described above.
        • In the example below, the "hello.py" is the name of the script that should be present in the <command-line> section. If using the ${PYTHON} macro in the cmd, the first line ("#!...") is ignored.
          <tool:hello.py>
          #!/usr/local/imdc/core/python/bin/imdc -OO 
          print "Hello world!"
          </tool>
    • Troubleshooting Command-line Probes
      • Turn on Detailed Logging: Help -> Diagnostics -> Detailed Logs, and check Command-Line Probes and Notifiers
      • Enter the host name/address of the server being tested in the IP Address Filter to minimize traffic.
      • Look at the entries in the Debug Log file to see what was sent and what returned.
      • It is often useful to see exactly what a device sends. Another tool, in addition to the detailed logging, is Rex's HTTP Viewer. This shows the full response from an HTTP server, and is available at http://www.rexswain.com/httpview.html
    • Review - Command-Line Probes
      • Examine the NOAA Temperature probe described in the Knowledge Base or retrieve it from http://download.dartware.com/contrib/probes/com.dartware.tool.noaa.txt. It shows the interesting parts of the Command-line probe
        • It has the standard InterMapper Probe format, with a type of "cmd-line"
        • It also has the <tool:noaa-weather.py> script embedded into the probe file itself.
        • It returns two variables: $temp, which is the temperature at the weather station, and the $loc, the description of that location.
        • It also shows how to retrieve arguments via a Python script, and how to use a couple interesting Python libraries.
      • Import the probe into InterMapper, and add a device. The IP address can be "localhost", because the script connects to the NOAA weather feed at its own web address. Enter a location as a parameter. "KLEB" is the Lebanon NH airport, right next to our offices.
      • Look at the Status Window. You can see how the variables in the <command-display> section ($temp, $loc) have been substituted there.
  • TCP Probes
    • All TCP Probes perform these steps:
      • Execute a "TCP script" written in the language defined in the Developer Guide
      • The script makes one or more TCP connections to the device and sends and retrieves data from it
      • The script collects data, performs calculations and evaluates the data against any supplied thresholds
      • The script then executes a "DONE" command which sets the devices status/severity
      • The values are formatted and displayed in Status Window
    • TCP Probe File Syntax
      • Sample Probe: examine the built-in HTTP probe, com.dartware.tcp.http.txt
      • Use "tcp-script" as the type in the <header>, as well as these sections
      • <script> section (Required) These tags enclose the statements that make up the TCP script.
      • <script-output> section (Optional) Specifies a template for the output that goes into the Status Window. It works like the other display sections.
    • Troubleshooting TCP Probes.
      • Turn on Detailed Logging: Help -> Diagnostics -> Detailed Logs, and check TCP Probe connections.
      • Enter the host name/address of the server being tested in the IP Address Filter to minimize traffic.
      • Look at the entries in the Debug Log file to see what was sent and what returned.
    • Review - TCP Probes
      • Open the built-in HTTP probe at: InterMapper Settings/Probes/com.dartware.tcp.http.txt or retrieve it from: http://download.intermapper.com/contrib/probes/com.dartware.tcp.http.txt
      • It has the standard <header> section, with "tcp-script" as the type.
      • It establishes a connection to the device's address (CONN #60), and if the connection doesn't succeed in 60 seconds, the device is marked as Down.
      • It sets an idle handler (WAIT #30 @IDLE) and if any operation times out (> 30 seconds), it transfers to the @IDLE label.
      • It sets a disconnect handler, and if an unexpect disconnect happens, it transfers to the @DISCONNECT label.
      • It next creates the full URL (fullURL) by starting with "http://" and concatenating the ${Host Name} parameter and the remote port and URL Path as needed.
      • It builds the HTTP request, sends it (using several SEND commands), then waits for responses. If none arrives, then the Idle or Disconnect handlers are invoked. If there is a bad status, the script exits with warnings (DONE WARN ...)
      • Finally, if the HTTP response has with a good status, the script checks the response to see if the desired string is present.
  • Resources and References
    • Developer Guide: This is the complete reference that describes InterMapper's under-the-covers operations. It describes how to create InterMapper Probes, how to modify the InterMapper Web GUI, and the InterMapper HTTP API.
    • Forum:  The Forum is a compendium of notes from active InterMapper customers and HelpSystems staff.
    • MIB Viewer Probe Builder: A MIB Viewer Probe shows all the variables that are supplied by a MIB. It automatically creates a full InterMapper SNMP probe that displays all the variables and tables. The page also has links to various SNMP resources.
    • SNMP Probe Builder: A web page that creates an SNMP probe using the names of the desired variables to view. This is a subset of the full capabilities of the MIB Viewer probe builder page.
    • On-Demand Table Builder: A web page that creates an on-demand table using the names of the table and the interesting columns. This is a subset of the full capabilities of the MIB Viewer probe builder page.
    • Mibble MIB Browser: Mibble is a Java application that reads MIB files and displays them in a readable format. Dartware has created a fork of the program that adds a few improvements to its ease of use. Download it, unzip it, and double-click the "MibbleBrowser.jar" file. Read the Tech Note to learn more about using Mibble.
    • Using InterMapper Probes - Part 1 and Working with InterMapper Probes - MIB Viewer Probes videos on Youtube.com. Also search for "InterMapper" to find related videos.
    • If you need to brush up on your Javascript skills, W3 Schools (http://w3schools.com/) is a good resource.
    • There are Knowledge Base articles for Python and Perl command-line probes at:http://www.helpsystems.com/intermapper/using-system-python and http://www.helpsystems.com/intermapper/simple-command-line-probe
    •  
  • Best Practices for Creating and Debugging Probes
    • InterMapper Probe files are text files. Line endings (Unix, Windows, Mac) are preserved, but are treated identically. It is useful to use a text editor that treats all line endings equally. Notepad++ (Windows) and TextWrangler (Mac) are free alternatives. There are many good choices for Linux.
    • A good name for the probe file is package.probe_name.txt, e.g., com.dartware.goodprobe.txt The package (com.dartware) will keep your probe files unique from other contributors, and probe_name should describe the probe itself. Always create your probe file names with a ".txt" suffix so they are easy to edit on any operating system.
    • When creating a new probe, it is often useful to clone an existing probe. If you do this, be sure to modify the package and probe_name in the <header> section right away, so InterMapper will keep the probes separate.
    • Use a text editor with fixed width font to make it easier to line up text in the probe file itself as well as in the display section of the Status Windows. It's a good practice to use spaces and tabs so the entries of the various <section>s are nicely aligned to help catch errors.
    • The InterMapper Settings / Probes folder contains InterMapper Probe files. It is safe to edit the probe directly (live) on the running server. The InterMapper server loads its probe files when it starts up, and does not check again until you use the Reload Probes command.
    • To reload probes, make the map editable, then use the keyboard shortcut (Ctl-Shift-R, or Cmd-Shift-R on Macintosh) or choose Monitor->Set Probe... and click the reload button at the lower left. Either technique reloads all probes, incorporates any changes you have made, and shows whether there were any errors during the reload process. You can view any errors encountered when reloading probes in the EventLog window.
    • After reloading probes, devices with modified probes are repolled immediately. Changes to the formatting of Status Windows may not always update the window right away, so it's a good idea to close and reopen them.
    • Using a Version Control System: Mercurial
      • There are no automatic backups of probe files on the InterMapper server, so it is useful to use a version control system. Mercurial has been used on a repository of the entire InterMapper Settings/Probes folder. InterMapper simply ignores the .hgignore and other meta-information files in its normal operation, while giving you the ability to revert to previous versions of the probe files.
      • On some operating systems (OSX, Linux), the InterMapper installer changes the permissions on the Probes directory so that it is only writable by the intermapper user. You may need to change the permissions on the Probes directory so that your normal login has read-write access before using a version control system.
      • The GUI programs MacHg (Macintosh) and TortoiseHg (Windows) have been used successfully. There is also a version of TortoiseHg for Linux,. Of course, the command-line versions of hg will work fine on all platforms.

Still have questions? We can help. Submit a case to Technical Support.

Last Modified On: December 10, 2016