This article explains how to troubleshoot:
- Missing events that were expected to be sent or logged to the expected Output.
- Missing Extensions in the event sent to the Output.
If you are not receiving expected events:
- Ensure that SIEM Agent been started. Use the following commands to start Central Administration and SIEM Agent:
Note: For PTSALIB/PSASTRMON, if you are using the command prompt with F4, be sure to change the Commit parameter to *Yes. (Press F9 to to change the Commit Parameter.)
- Ensure that the Output has been attached to the Event Source:
- From the Main Menu, choose 3 to open Work with Outputs.
- Ensure the Output exists and is active. If it is not active, use option 6 to activate.
Note: Correct Output settings will be confirmed later in this procedure if required.
- Return to the Main Menu and choose 1, then 2 for the Event Source.
- Press F8 to ensure the correct Output is attached to the Source as its Default Output. If not, press F6 to attach the Output as a Default Output.
- Be sure to confirm all changes.
- Ensure the Active flag is set to 1 for the Event Source, Event Description, and Subtype:
- From the Main Menu, choose 1.
- Choose 2 for the Event Source.
- Set Active to 1 (if it is not already) and press Enter.
- Choose 8 for the Event Source to open Work with Event Descriptions.
- Use 6 to toggle inactive events from 0 (inactive) to 1 (Active).
- Subtypes, if they exist, must be active as well in order to be sent to the Output. Choose 8 for the Event Description (Journal Code/Entry Type pair) to open Work with Event Subtypes. Use 6 to toggle inactive subtypes (0) to Active (1).
- Review Rules to confirm the expected Output is configured:
- From the Main Menu, choose 1, then 9 for the Event Source to show the Event Descriptions.
- Choose 9 for the event. If Rules are attached:
- Use 8 to confirm the Conditions of the Rule are correct.
- Use 2 to confirm the Rule settings are correct. If Rule Output is set to “None” the Rule is preventing the event from being sent to the Output.
- Use F8 to open Work with Attached Outputs, where you can verify the Output attached is the correct one and set to Active (1).
- Return to Work with Event Descriptions. If the event is a Subtype, use 8 to show the Event Subtypes for the Event Description. Go through the same steps as above for the Rules attached to the Event Description.
- If you made any changes, return to the Main Menu, then commit the changes by selecting Option 82 "Work with Utilities", then option 1 "Commit configuration changes". Then press ENTER to confirm.
If at this point the issue has not been resolved, proceed to the next section.
For Events Expected from the Security Audit Journal
Powertech SIEM Agent for IBM i can only forward Events from the security audit journal if the operating system logs them there. If you are expecting to send Events from the Security Audit Journal, check the following:
- The audit journal has been set up, including journal receivers and configuration of system values such as QAUDCTL.
- The audit level includes the categories corresponding to the type of event that you want to capture.
- The audit level is configured by means of the QAUDLVL system value. If QAUDLVL contains the value *AUDLVL2, the setting of the QAUDLVL2 system value is applied in addition to the setting of the QAUDLVL system value.
- For example, the audit level must contain the value *AUTFAIL ("authorization failure") in order for T:PW ("password failure") Events to be logged in the security audit journal. For more information about which QAUDLVL/QAUDLVL2 settings correspond to which types of security audit journal entries, see the IBM i Security Reference.
- The events have actually occurred. For instance, password failure events are only logged in the security audit journal if a user or outside process tried to authenticate against this system with an incorrect password (or a non-existing user profile name).
If at this point, the issue has not been resolved, proceed to the next section.
For Output Intended for a SIEM (Output Type *NETWORK)
- Determine the following information:
- Address (IP or name) of SIEM. (Note: Version 4.0 does not support the use of the FQDN.)
- Protocols the SIEM accepts (UDP, TCP, TLS).
- Port the SIEM accepts messages at.
- Check firewall settings:
- Does a firewall control the network traffic between the local system and the SIEM? If yes, ensure the firewall allows the events to be received by the SIEM. The protocol and (destination) port to open in the firewall must match those configured on the Output.
- Ensure the correct address has been specified on the Location field of the Output:
- If a numerical IP address specified, double-check that the IP address is correct.
- If a name is specified, use a PING from that IBM i to check that the name is resolved correctly. (The PING may not be replied to, but the name must be resolved to an IP address.)
- Ensure that the SIEM is configured to accept messages:
- From this IBM i (source address).
- Using the protocol configured on the Output.
- Using the port configured on the Output.
- Local-copy check:
- Add an Output of type *STREAM.
- Ensure that the directory in which the Stream File should reside already exists.
- Ensure that user profile PTUSER has *CHANGE authority to the directory.
- Add the new Output to the Default Output of Event Source.
- Confirm changes.
- Provoke the event.
- Observe if a message reflecting the new message/event is added to the local log:
- If yes: Issue is likely to be firewall, network, SIEM, or mismatch between SIEM settings and configured Output pointing at the SIEM.
- If no: Issue is more likely to be in the Event Source, Event Description, or Rules.
For Output Intended to Be Sent to a Stream File (*STREAM)
- Ensure that the directory in which the Stream File should reside already exists.
- Ensure that user profile PTUSER has *W(RITE) authority to the directory.
For Output Intended to be Sent to a Message Queue (*MSGQ)
- Ensure that the message queue already exists.
- Ensure that user profile PTUSER has *CHANGE authority to the message queue.
- Ensure that the message queue is not both full and set to not *WRAP.
If Events Are Received but Extensions are Missing
When output does not contain Extensions that you defined, the Output can be configured to use the legacy *CEF or SYSLOG Message Type. This message type does not allow Extensions to be included in the message. Use an output with the *MODERN Message Type instead.
Resolving with SIEM Agent’s Tracing Feature
If the above steps did not resolve the issue:
- Enable tracing:
PSATRCSIEM MONITOR(*MAIN) TRACE(*START) PATH('/tmp/PTSA-trace-main')
PSATRCSIEM MONITOR(name of Event Source) TRACE(*START) TYPE(*SOURCE) PATH('/tmp/PTSA-trace-source01')
PSATRCSIEM MONITOR(name of Output) TRACE(*START) TYPE(*OUTPUT) PATH('/tmp/PTSA-trace-output01')
- Recreate the issue:
- For example, if you are trying to forward or log audit journal events, cause the audit journal entry to be deposited (created). To provoke a TAD (System Values Modification) event, for example, modify an object’s audit settings.
- End tracing:
PSATRCSIEM MONITOR(*MAIN) TRACE(*STOP)
PSATRCSIEM MONITOR(name of Event Source) TRACE(*STOP) TYPE(*SOURCE)
PSATRCSIEM MONITOR(name of Output) TRACE(*STOP) TYPE(*OUTPUT)
- Transfer the trace files to your PC:
- Copy the following files to your PC. You can transfer them using any method available, such as FTP or the file server.
- Create a support case through the Community Portal. Provide the following information:
- Description of the issue.
- Screenshots of the configuration of the Event Source, Event Description, Subtype (if one exists), Rules attached to the Event Description, Rules attached to the Subtype.
- The trace files.
- Provide the trace files created and transferred in the preceding steps as attachments to the support case.
Checking for Outgoing Traffic
If Events are configured to be sent to a remote SIEM, but are not received by the SIEM, the following steps enable you to check if the Events are being sent out.
- In the Output, note the IP address and port, and whether the connection uses UDP, TCP, or TLS.
- Provoke the Event.
- Run the command NETSTAT.
- Select option 3, Work with IPv4 connection status.
- If the Output is configured to use TCP or TLS, look for a connection with the attributes below. Note that you may need to press F11 to change the columns displayed, or to display details for a connection.
- The Remote Address of a connection equals the remote address configured for the Output (or—if a name was configured—what that name resolves to).
- The Remote Port of the connection equals the port configured on the Output.
- (Connection) User is PTUSER.
After pressing F11 to display the job user and the Byte counts:
The highlighted line is for a TCP connection to a SIEM, over port 601.
If the Output is configured to use UDP, look for a connection with the attributes below. Note that you may need to press F11 to change the columns displayed, or to display details for a connection.
Local host name . . . . . . . . . . . :
Local internet address . . . . . . : *
Local port . . . . . . . . . . . . : (portnumber is 1024 or higher) = high and random)
Associated user profile . . . . . . . : PTUSER
Bytes in . . . . . . . . . . . . . . : 0
Datagrams in . . . . . . . . . . . : 0
If you display the job for this connection, the job should be named PSAMGRMON.
If SIEM Agent is sending out UDP traffic, you will see the “Bytes out” and “Datagrams out” counts for the connection increase with each Event that it sends out.
Note: The connection only exists after at least one event has been sent out by SIEM Agent.
To see which server is sending data:
- "Display jobs" for the connection.
- Note the job's job number name.
- Submit WRKACTJOB JOB(PSAMGRMON).
- Press F11 twice so you see the job number.
- Identify the job/s by the job number.
- Then press F11 again to display the job function.
- The job function, minus the "USR-" prefix, will show you the name of the Output.
If data is being sent out by the connection, but not received by the SIEM, please check the items above (IP address, port, firewall, SIEM settings).
For Output Using the JSON Format
Data will only be sent if Extensions have been defined on the Event Description or on Rules. This differs from other Formats, which support default fields. Outputs using the JSON Format—using a Format that has Message Style *JSON—only send data to an Output if the Event is processed by an Event Description or Rule that defines custom Extensions.