Palo Alto Cortex Data Lake

Palo Alto Networks Cortex Data Lake (Cortex Data Lake) provides cloud-based log storage that uses artificial intelligence to analyze all your data at once. You can achieve a unified experience by configuring your Palo Alto products to send all data to Cortex Data Lake.

You can learn more about Cortex Data Lake by visiting the product website at: https://www.paloaltonetworks.com/cortex/cortex-data-lake.

The Cortex Data Lake event source allows InsightIDR to parse the following log types:

  • Web Proxy
  • Firewall
  • Ingress Authentication
  • VPN session
  • IDS
  • Hostname to IP
  • Advanced Malware
  • Virus Infection

To set up Cortex Data Lake, you’ll need to:

  • Review the requirements.
  • Add the Palo Alto Cortex Data Lake event source in InsightIDR.
  • Forward logs from Cortex Data Lake to NXLog to InsightIDR.
  • Verify the event source configuration works.

Phase 1: Review the requirements

Palo Alto requirements

For information about Palo Alto products that use Cortex Data Lake and their requirements, see Palo Alto’s documentation at: https://docs.paloaltonetworks.com/cortex/cortex-data-lake/cortex-data-lake-getting-started/products-that-use-cortex-data-lake-container/products-that-use-cortex-data-lake

NXLog requirements

  • Access to the host where you will install and configure NXLog. You may use your InsightIDR Collector as the host that will be running NXLog.
  • If you have malware detection software installed on your NXLog host, be sure that the software will not block NXLog from operating.
  • Ability to encrypt logs for transfer between Cortex Data Lake and NXLog.
  • A public hostname that points to the public IP address of the NXLog server and matches the names associated with the certificate.
  • If NXLog is running on the same host as your Collector, verify that any local Firewall services or malware detection software allows NXLog to send logs to your Collector. You can read more about these requirements in our NXLog documentation.

Phase 2: Add the Palo Alto Cortex Data Lake event source in InsightIDR

Would you like to send encrypted logs from NXLog to the Collector?

If you want to encrypt the logs from NXLog to the Insight Collector, select the Encrypted checkbox when setting up Cortex Data lake in InsightIDR, and download the Rapid7 certificate. You will need this certificate when configuring NXLog.

Set up Cortex Data Lake in InsightIDR

  1. From the left menu, select Data Collection.
  2. When the Data Collection page appears, click the Setup Event Source and choose Add Event Source.
  3. Click the Cloud Services category under Security Data.
  4. Choose your Collector.
  5. From the Event Source Type dropdown, choose the Palo Alto Networks Cortex Data Lake event source.
  6. Choose the time zone that matches the location of your event source logs.
  7. If you are sending events other than alerts and want to view them in Log Search, select the Send unparsed logs checkbox.
  8. You can specify a Default Domain or Add a New Domain if needed.
  9. In the Port field, specify a port number.
  10. Choose a Protocol.
  11. (Optional) Check the box to Encrypt your logs using NXLog before sending to InsightIDR.
  12. Click Save.

Phase 3: Forward logs from Cortex Data Lake to NXLog to InsightIDR

Cortex Data Lake’s encryption requirements require you to set up a machine capable of receiving logs from Cortex Data Lake and forwarding them to the InsightIDR Collector. Rapid7 recommends using syslog by way of NXLog to receive logs from Cortex Data Lake.

Enable communication between Cortex Data Lake and your syslog receiver; follow the steps provided by Palo Alto to allow an inbound TLS feed to your NXLog host. To forward logs from Cortex Data Lake, follow Palo Alto’s documentation at: https://docs.paloaltonetworks.com/cortex/cortex-data-lake/cortex-data-lake-getting-started/get-started-with-log-forwarding-app/forward-logs-from-logging-service-to-syslog-server

To create a rule that allows only this event source’s traffic through your firewall to your designated host, use a network address translation (NAT) to map a public IP address to your Collector’s private IP address. Follow the documentation for your Palo Alto external firewall to create the NAT at: https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/networking/nat.

Key items to note when setting up log forwarding:

  • You must configure a log filter to select the logs you want to forward or Cortex Data Lake will not forward any logs at all.
    • Select all of the types of logs that you want to forward, and send them to the same event source in InsightIDR, which is the one you will be setting up for Cortex Data Lake events.
  • For the syslog facility, you may send the logs in CSV (recommended), LEEF, or CEF format. If you are asked to configure a field delimiter, use the default delimiters for these facility types.
  • InsightIDR does not require client authentication. However, you will need to configure the syslog profile to use the certificate file and key file that you previously obtained from Cortex Data Lake.
  • You will not be able to test connectivity between Cortex Data Lake and NXLog until after you have configured NXLog and started the NXLog service.
  • For the listening port, specify the port you will use to forward the syslog to NXLog. In the NXLog configuration, you will configure NXLog to listen on the same port.

Configure NXLog to collect encrypted syslog

There are 4 basic steps to configuring the collection of encrypted logs with NXLog.

  1. Install NXLog.
  2. Configure Cortex Data Lake to send encrypted logs to NXLog.
  3. Configure NXLog to decrypt logs and forward them to your Collector.
  4. Verify log collection and troubleshoot issues.

Step 1: Install NXLog

  • Typically NXLog is installed on your Collector, but you can use a different host if you prefer.
  • For Windows operating systems, follow our product documentation to install NXLog.
  • For Linux operating systems, follow our blog to install NXLog.

Step 2: Configure Cortex Data Lake to send encrypted logs to NXLog

Step 3: Configure NXLog to decrypt logs and forward them to your Collector

  1. If the NXLog service is running on your NXLog host, stop the service before continuing.
  2. Replace the default NXLog.conf file with the sample configuration file.
  3. Copy your certificates into a new folder on your NXLog host, or use the \NXLog\cert directory, to store your certificates.
  4. Edit these NXLog.conf fields to match your environment:
    • ListenAddr
    • CAFile
    • CertFile
    • CertKeyFile
    • Host
    • Port
  5. Start the NXLog service.

For additional information on configuring NXLog, refer to our blog.

NXLog will forward the logs to the Collector using UDP by default. If you want to send the logs using encrypted syslog (because you selected Encrypted on the event source setup page and downloaded the Rapid7 certificate), you must use the om_ssl module instead of om_udp in the Output section of your nxlog.conf file. Review our documentation on sending encrypted logs.

View a sample NXLog configuration file to ensure you have the correct Output section in your nxlog.conf file.

Example output module for encrypted syslog

1
<Output out>
2
Module om_ssl
3
#IP of the IDR Collector
4
Host 10.1.1.1
5
#Port, must match what is configured for the event source
6
Port 6515
7
#Exec to_json();
8
</Output>

Example output module for unencrypted (standard) syslog

1
<Output out>
2
Module om_tcp
3
#IP of the IDR Collector
4
Host 10.1.1.1
5
#Port, must match what is configured for the event source
6
Port 6515
7
#Exec to_json();
8
</Output>

To obtain an entire NXLog.conf file, see the sample NXLog configuration file.

Step 4: Verify collection and troubleshoot issues

  1. Check your NXLog.log file for errors. The audit log will be located where you specified in the NXLog.conf file. If you can't find your NXLog.log file, open the NXLog.conf file to find where the log is being created, which is indicated in the LogFile variable. The default location is: LogFile %ROOT%\data\nxlog.log
  2. To determine if logs are flowing to InsightIDR, navigate to Data Collection -> Event Sources and find the Palo Alto Cortex Data Lake event source in the list. Select the View Raw Log button on the Palo Alto Cortex Data Lake event source to see if any logs are listed. If the View Raw Log page is empty, then the Collector has not received any logs.

For additional information on verifying collection and troubleshooting NXLog issues, including setting up and troubleshooting for a Linux environment, refer to our blog.

Phase 4: Verify the configuration

Complete the following steps to view your logs and ensure events are making it to the Collector.

  1. On your new Cortex Data Lake event source, click View Raw Log. If you see log messages in the box, this shows that logs are flowing to the Collector.
  2. In the left menu, click Log Search.
  3. Select the applicable log sets and, within the sets, select the log names. The log name will be the event source name or Palo Alto Networks Cortex Data Lake if you did not name the event source. Palo Alto Networks Cortex Data Lake logs flow into the Palo Alto Networks Cortex Data Lake log set.

If you see log messages when you select View Raw Log on the event source but do not see any log messages in Log Search after waiting for a few minutes for them to appear, then your logs do not match the recommended format and type for this event source.

Supplemental materials

Sample NXLog configuration

Review the NXLog reference manual about the configuration options at: https://docs.nxlog.co/userguide/configure/index.html.

Sample configuration file

Set the ROOT to the folder your NXLog was installed into, otherwise it will not start.

1
define ROOT C:\Program Files\nxlog
2
define CERTDIR C:\Program Files\nxlog\cert
3
Moduledir %ROOT%\modules
4
CacheDir %ROOT%\data
5
Pidfile %ROOT%\data\nxlog.pid
6
SpoolDir %ROOT%\data
7
LogFile %ROOT%\data\nxlog.log
8
9
<Extension cef>
10
Module xm_cef
11
</Extension>
12
<Extension syslog>
13
Module xm_syslog
14
</Extension>
15
<Input ssl>
16
Module im_ssl
17
ListenAddr 0.0.0.0:16514
18
CAFile %CERTDIR%/datalake.cert
19
CertFile %CERTDIR%/plzwork.crt
20
CertKeyFile %CERTDIR%/plzwork.key
21
Exec parse_syslog();
22
</Input>
23
<Output udp_output>
24
Module om_udp
25
Host 127.0.0.1
26
Port 16515
27
</Output>
28
<Route 1>
29
Path ssl => udp_output
30
</Route>

Sample logs

The sample log shown is in CSV format with a syslog header.

1
<190>Feb 06 10:33:21 R7-5050 1,2020/02/06 10:33:21,0009C101184,THREAT,wildfire,0,B2020/02/06 10:33:21,149.163.216.66,31.14.86.8,0.0.0.0,0.0.0.0,lan-wan-allow-all,,,rapid7-base,vsys1,lan-unipa,wan,ethernet1/24.5,ethernet1/23,syslog-forward-profile,2020/02/06 10:33:21,34513570,1,58006,443,0,0,0xf000,tcp,alert,"rapid7.com/",(9999),social-networking,informational,client-to-server,4287654456,0x0,Italy,Ireland,0,,0,,,0,,,,,,,,0,0,0,0,0,vsys-unipa,PA-5050,,,,,0,,0,,N/A,unknown,AppThreat-0-0,0x0