Troubleshooting

It is important to be able to debug and troubleshoot when developing plugins.

UI

Errors and debug information can be found in the UI on the job page by viewing the Error and Log tabs.

Logs

The engine and web services contain logs that may help diagnose issues.

Enable debug logging to get the most out of them by adding the following line to /opt/komand/etc/web.conf and /opt/komand/etc/engine.conf:

KOMAND_LOG_LEVEL=debug

Then restart the services:

 
1
systemctl restart komand-engine
2
systemctl restart komand-web

Now, investigate the logs:

 
1
tail -f /var/log/komand/web.log
2
tail -f /var/log/komand/engine.log

Containers

You can view logs of running containers by interacting with Docker. This is helpful in cases where no jobs are created by a trigger in the UI. To do this, you will need to be on an InsightConnect orchestrator or on a Komand host with a privileged account such as root.

Obtain container ID:

 
1
root@localhost ~]# docker ps
2
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS
3
NAMES
4
ba7cdcd752fb        komand/twitter:1.0.0   "/usr/local/bin/koman"   12 minutes ago      Up 12 minutes

View logs in realtime from running container by ID:

 
1
[root@localhost ~]# docker logs -f ba7cdcd752fb
2
INFO:root:Using dispatcher config: {u'url':
3
u'http://127.0.0.1:8000/v2/workflows/0a64d852-8384-43f3-b29e-1be835b3b018/events?internal=1&api_key=2981d3fb-d7ca-47d7-9a05-ceb2123271d2'}
4
INFO:root:Connect: Creating Twitter client.
5
INFO:root:Right about to create API client
6
INFO:root:API Client created
7
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com
8
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com
9
INFO:root:Connect: Connected! ID: 778783652834582529, Username: Rusty Shackleford, ScreenName: DoctorPeepers12
10
INFO:root:OpenCacheFile: /var/cache/triggers_twitter_messages exists, returning it
11
INFO:root:Run: Iterating main loop
12
INFO:root:OpenCacheFile: /var/cache/triggers_twitter_messages exists, returning it
13
INFO:root:Run: Cached id is 787384615166152708.
14
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): api.twitter.com
15
INFO:root:Get Messages: Got 0 messages.
16
INFO:root:Get Messages: Next sleep duration set to 360 seconds.
17
INFO:root:Run: No new messages. Sleeping 360 seconds.

You can also debug a running plugin container live using the side-car container method. We have prepared a debugging container with basic tools that you can pull down with

# docker pull komand/python-debug-slim-plugin

Once you have the debugging image, find the container ID or name of a running plugin container that you would like to debug:

sh
1
# docker ps
2
e0a52e7da434        rapid7/kafka:1.0.2                       "/komand/plugins/bin…"   4 weeks ago         Up 4 weeks          0.0.0.0:10002->10001/tcp   rapid7_kafka_1.0.2_action

Next, we run the debugging container with settings to attach it to the process and network namespace of the running plugin container. And finally, we start calling debugging tools on the plugin process:

sh
1
# docker run --rm -it --privileged --pid=container:rapid7_kafka_1.0.2_action --net=container:rapid7_kafka_1.0.2_action komand/python-debug-slim-plugin bash
2
bash-4.4# ps
3
PID   USER     TIME  COMMAND
4
    1 root      1:03 /komand/plugins/bin/kafka http
5
   13 root      0:00 bash
6
   19 root      0:00 ps
7
bash-4.4# strace -p 1
8
strace: Process 1 attached
9
futex(0xb39f18, FUTEX_WAIT, 0, NULL
10
...
11
bash-4.4# netstat -plunt
12
Active Internet connections (only servers)
13
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
14
tcp        0      0 :::10001                :::*                    LISTEN      1/kafka

You can see that we’re able to access the processes and network of the attached plugin container.

UI

Bugs can happen in the UI and cause unexpected results or block your work. To check for bugs, open your browser’s developer tools to look for errors. If found, report them to the InsightConnect team in Slack.

File Uploads

Testing file uploads via API trigger can be done with the following shell function:

 
1
$ declare -f postfile
2
postfile ()
3
{
4
    local var="$1";
5
    local file="$2";
6
    local url="$3";
7
    local token="$4";
8
    local argc="$#";
9
    [[ $argc -ne 4 ]] && {
10
        printf "postfile <var> <path/file> <url> <key>\n" && return 0
11
    };
12
    b64=$(base64 ${file});
13
    json=$(echo -n '{'\""${var}"\"': '\""${b64}"\"'}');
14
    echo $json | eval curl -H \"Authorization: $token\" -H \"Content-Type: application/json\" -d @- $url
15
}

Usage, where < var > is the name of the API trigger variable for which the file will be the value (script only supports one currently):

 
1
$ postfile
2
postfile <var> <path/file> <url> <key>

Example: $ postfile pcap ~/google-http.pcap http://127.0.0.1:8888/v2/workflows/6535b9ba-1930-4c56-a5e2-1904b048a4ee/events bae09ddd-a593-4113-96a6-7cedb816cf1a