Troubleshoot a Workflow

Workflow failures in InsightConnect can originate in many places, ranging from an error in the workflow to a temporary outage in a service that the workflow relies on. Putting the error in context is a necessary step to determine the source of the failure.

Troubleshooting Overview

InsightConnect workflows can be triggered in 3 ways:

• Through the Test button in the builder
• Manually on the Active Workflows page
• In response to the workflow's trigger

In the latter two cases (by a workflow's trigger or manually from the Active Workflows page), a job will be created on the Jobs page. However, when testing a workflow from the builder, a job record is not created.

The data available to troubleshoot is the same, regardless of how the workflow is triggered, but workflow tests that run from the builder are not persisted, so the page must be kept up during troubleshooting so you can reference the data.

• The workflow was working in the past, but now every job is failing
• The workflow isn't being triggered (no jobs being created)
  • InsightIDR and InsightVM Platform Triggers
  • Plugin Triggers (including InsightVM plugin-based triggers)
• The workflow is generating a failed job, either continually or intermittently

The workflow was working in the past, but now every job is failing

If the workflow hasn't changed, but it has started failing persistently, the problem is likely outside the workflow. Take the following steps:

1. Check that the orchestrator is healthy under Settings > Orchestrators. If the orchestrator is unhealthy, see the Troubleshoot an Orchestrator documentation for further instructions.
2. Check the credentials the workflow is using. Run a connection test under Settings > Plugins & Tools for all connections that the workflow utilizes.
3. Follow the instructions for troubleshooting a failing workflow.

The workflow isn't being triggered (no jobs being created)

If a workflow isn't triggering at all, something must be wrong with the trigger. Depending on the type of trigger, the troubleshooting steps vary.

InsightIDR and InsightVM Platform Triggers

InsightIDR and InsightVM triggers are configured in their respective products. The first thing to check is that the trigger is properly configured in those interfaces. This does not include the InsightVM plugin trigger. For the InsightVM plugin trigger, follow the instructions in the Plugin Triggers section to troubleshoot.

If a workflow is triggering but the expected data is missing, it means that InsightIDR or InsightVM isn't sending the data to InsightConnect. It is likely that the alert triggering the workflow is misconfigured.

Plugin Triggers (including InsightVM plugin-based triggers)

Plugin-based triggers run on the orchestrator. They connect to a remote system or service and monitor for specific conditions. When the conditions are met, they trigger the workflow.

Steps to troubleshoot a plugin-based trigger:

1. Check the health of the orchestrator under Settings - if the orchestrator isn't showing as "healthy" then there is an issue with the orchestrator itself. Use the Troubleshoot an Orchestrator documentation to get the orchestrator into the healthy state.
2. Run a connection test on the plugin using the three-dot menu on the Plugins & Tools page under Settings. Resolve any errors indicated.
3. Verify that the trigger configuration in the workflow is correct. The error is often due to the configuration causing the trigger to monitor the wrong data (such as the wrong mailbox) or discard the event you expect it to trigger on.
4. Connect to the orchestrator (using SSH or the virtual machine console).
5. The trigger runs as a Docker container on the orchestrator. Run sudo docker ps -a|grep trigger to see the active trigger containers.
6. Identify the trigger based on the data displayed (the wider the window, the easier this will be). If you are unable to identify the trigger, try deactivating the workflow, wait about 2 minutes, and then reactivate it. This will restart the trigger. You'll see this reflected in the container's uptime when listing the active triggers.
7. Use the container ID (the first column in the output) to examine the logs for the trigger with sudo docker logs <container id>. The container logs are the diagnostic output from the trigger. You can add the --follow flag to watch them in real-time (sudo docker logs --follow <container id>)

If the trigger container is not present, refer to the Troubleshoot an Orchestrator documentation and contact Rapid7 Support.

The workflow is generating a failed job, either continually or intermittently

Workflows are built with certain assumptions: the trigger contains certain data, it can look up data in other systems and services, etc. When a job is intermittently generating a failed job, it means these assumptions are not being met. Usually, but not always, this means the workflow must be modified to account for the condition that's causing it to fail.

1. Under Jobs, select the job that failed.
2. Select the All Outputs tab.
3. Find the failed step in the job.
4. Examine the Input tab of the step output. Verify that the input is both what you expect and what the plugin expects. A common problem is a missing or malformed input.
5. If the input is incorrect, examine the workflow and determine where the input is coming from. Incorrect input usually means that a previous step is returning unexpected data.
6. Examine the Logs tab of the step output. This log contains diagnostic information related to the failure, such as error messages returned from the underlying service.
7. If the log indicates permission issues, run a connection test on the Settings > Plugins & Tools page and verify that the credentials are correct. If the connection test succeeds but the step is still failing due to permissions, it is likely the plugin doesn't have the correct permissions to perform the action.
8. If the log indicated timeout issues, there could be a service disruption of the underlying service or the orchestrator is experiencing connection difficulties with the underlying service.

If you do not see any errors, or there are errors you cannot fix, please take a screenshot of your Inputs and Logs and reach out to Rapid7 Support.