The following steps will guide you through identifying and resolving common issues. Follow the steps in sequence to troubleshoot effectively. If the issue persists even after completing these steps, consider reaching out to Support for further assistance.

Step 1: Enable Flags to Verify SysLog Traffic

  1. Login to the gateway.
  2. Execute the gcli command.
  3. Execute the following commands to enable flags for additional logging:
    flag add syslog.log on 40
    loglevel set com.vistara.gateway.syslog.SyslogEventProcessor DEBUG 60
    loglevel set com.vistara.gateway.syslog.SyslogMessageHandler DEBUG 60
  4. Execute exit to leave gcli.
  5. Replace the SysLog server IP address and execute the command below:
    tail -100f /var/log/app/vprobe.log | grep “UDP : Ip : <Ipaddress>”
  6. Send the SysLog event to the gateway and verify whether the log message appears in the gateway logs.
  7. If you do not find any log messages, proceed to Step 2 and capture the packets.

Step 2: Capture Packets to Verify SysLog Traffic

  1. Launch another gateway console and execute the following tcpdump:
    tcpdump -i any "udp port 514" -s 3000 -w /tmp/syslog.pcap
  2. Send the SysLog event to the gateway and verify whether the packet count increases.
  3. If the packet count increases, press Ctrl+C to stop packet capture.
  4. Download the pcap file and open it in Wireshark to verify the packet details.
  5. If no packets are captured or the required packets are missing, verify the configuration on the SysLog server.
  6. If valid packets are captured, continue to Step 3 to verify the gateway configuration.

Step 3: Verify Pushed SysLog Configuration

  1. Execute the gcli command.
  2. Run the following command to print cached SysLog configurations:
    syslogcache print
    Sample Output:
    Syslog Cache Collection is written to file: /var/log/app/tmp/syslogcache-1720088523726.json
  3. Execute exit to leave gcli.
  4. Display the contents of the generated file using the command:
    cat /var/log/app/tmp/syslogcache-1720088523726.json
  5. Verify that the configuration rules are correctly present in the file. Specifically, check the IP address, severity, facility of the event, and regex pattern in the rules. If incorrect or missing, create or update the configuration profile/rules accordingly.

Step 4: Check if Rules are Dropping Events

  1. Login to the gateway.
  2. Execute the gcli command.
  3. Enable additional logs using the following commands:
    flag add syslog.log on 40
    loglevel set com.vistara.gateway.syslog.SyslogEventProcessor DEBUG 60
    loglevel set com.vistara.gateway.syslog.SyslogMessageHandler DEBUG 60
  4. Execute exit to leave gcli.
  5. Run the following command to observe logs:
    tail -100f /var/log/app/vprobe.log
  6. If you find log messages similar to the one below, it indicates that events are being skipped by rules:
    SyslogEventProcessor#96: Skipped:Severities. IP : 10.212.0.7, severity: 5, Profile: ABC Test syslog profile
    If alerts are needed for these events, review and update the configuration profile.

Step 5: Check Gateway Alert Throttling

  1. Understand that OpsRamp Gateways apply alert throttling to reduce alert noise and optimize processing.
  2. By default, the Gateway limits 4 alerts per 10 minutes for each unique combination of:
    • Device IP
    • Metric name
    • Component name
      (Collectively referred to as deviceip_metric_component key)
  3. Once the threshold is reached:
    • Additional alerts for that key are not sent to the OpsRamp cloud until the 10-minute interval resets.
    • After the interval resets, the next alert will be:
      • Appended with “Attenuated Alert:" in the alert subject
      • Marked as CRITICAL
  4. In addition to gateway-side throttling, the OpsRamp cloud processes only one alert per key per minute. If multiple alerts are received within the same minute, only one is retained.
  5. If your SysLog events are not appearing as alerts in the platform:
    • Review /var/log/app/vprobe.log to determine if throttling messages are logged.
    • Confirm whether multiple alerts were generated for the same key within a short duration.
  6. If needed, you can customize the SysLog monitor rule to define unique metric and component values, which helps distribute alerts across different keys and avoid hitting the throttling limit.
  7. There are no specific prerequisites to customizing these fields, and such changes do not affect the throttling behavior unless explicitly reconfigured with Support assistance.
  8. If behavior persists or seems abnormal, capture Gateway logs and contact OpsRamp Support.