Log4Net

Logging To Log Management (InsightOps)

Log Management (InsightOps) provides a plugin for log4net.

To configure your application to log to Log Management (InsightOps), you will need to perform the following tasks:

• Create a Log Management (InsightOps) account.
• Create a host and log to receive your log data.
• Adding the Plugin library and the appropriate Log Management (InsightOps) Appender libraries to your application.
• Configure the Plugin and the Log Management (InsightOps) appender in your application.
• Send log messages from your application.
• These steps are outlined in further detail below.

Setup

The easiest way to add the log4net and the Log Management (InsightOps) Target libraries to your application is to install the R7Insight.Log4net Nuget package . This package will install the Log Management (InsightOps) Target library and will also automatically install the log4net package as a dependency.

If you would rather install the Log Management (InsightOps) appender manually, you can download the complete code in this GitHub repository, compile the R7InsightLog4net Visual Studio project within it into a DLL file, and then reference this file in your application. If you choose this option you must install log4net yourself.

Configuring log4net and the Log Management (InsightOps) Appender

General log4net configuration is beyond the scope of this readme. Please refer to the Configuration section of the log4net manual for details on how to configure log4net.

log4net allows log messages to be sent to multiple destinations. In log4net terminology, such an output destination is called an appender. Appenders must implement the log4net.Appenders.IAppender interface. The Log Management (InsightOps) Appender library provides such an appender component that is specifically designed to send log messages to Log Management (InsightOps) in an efficient manner.

The Log Management (InsightOps) appender is configured and added to your log4net configuration in the normal way using an <appender> element:

<appender name="InsightAppender" type="log4net.Appender.InsightAppender, R7Insight.Log4net">
    ...
</appender> 

The Log Management (InsightOps) appender has two categories of settings that are configured somewhat differently:

• Logging settings
• Log Management (InsightOps) credentials

log4net Logging Settings

Logging settings determine how the appender operates, and are specified as child elements of the <appender> element. The Log Management (InsightOps) appender supports the following configuration settings:

• Level: The lowest Log4net logging level that should be included. All log messages with a logging level below this level will be filtered out and not sent to Log Management (InsightOps).
• Debug: Set to true to send internal debug messages to the Log4net internal logger.
• UseSsl: Set to true to use SSL to send data to Log Management (InsightOps) (see below for more information).
• Layout: The layout used to format log messages before they are sent to Log Management (InsightOps). See the Configuration section of the Log4net manual for more information on configuring layouts. Here is an example of an appender configuration that works well for Log Management (InsightOps):

   <appender name="InsightAppender" type="log4net.Appender.InsightAppender, R7Insight.Log4net">
        <Debug value="true" />
        <Region value="eu" />
        <UseSsl value="true" />
        <Token value="PLEASE PUT TOKEN HERE" />
        <layout type="log4net.Layout.PatternLayout">
          <conversionPattern value="%d{ddd MMM dd HH:mm:ss zzz yyyy} %logger %: %level%  - %m"/>
        </layout>
     </appender>

Next you need to place the following line in your AssemblyInfo.cs file: For web applications use:

[assembly: log4net.Config.XmlConfigurator(ConfigFile="Web.config", Watch = true)]

For Console apps use:

[assembly: log4net.Config.XmlConfigurator(ConfigFile="App.config",Watch=true)]

log4net Log Management (InsightOps) Credentials

Log Management (InsightOps) credentials determine to which host and log your log messages are sent. The following settings constitute the Log Management (InsightOps) credentials:

Token: The unique token GUID of the log to send messages to. This applies when using the newer token-based logging.

Therefore, the Log Management (InsightOps) credentials can be specified more flexibly than the configuration settings. You have three options:

• Specify the credentials as child elements of the <appender> element (if you don’t need the added flexibility).
• Specify the credentials as settings in the <appSettings> element in your App.config och Web.config file.
• Specify the credentials as Windows Azure role configuration settings in your cloud service project (only applicable when running your application as a cloud service in Windows Azure).

The Log Management (InsightOps) appender uses the CloudConfigurationManager class internally to read the credential values. This class looks for each credential value in the following order:

• If the value exists as a Windows Azure role configuration setting, that value is used.
• Otherwise if the value exists as a setting in the <appSettings> element in your App.config och Web.config file, that value is used.
• Otherwise if the value exists as a configured child element of the <appender> element, that value is used.
• If the value was not found in any of these locations, errors are logged to the Log4net internal debug log and logging to Log Management (InsightOps) will fail.

Here is an example of how to specify the credentials in the <appender> element:

<appender name="InsightAppender" type="log4net.Appender.R7InsightAppender, R7Insight.Log4net">
    <token value="bb61600f-f766-451e-b55f-9204f536a79f" />
    ...
</appender>

Here is an example of how to specify the credentials in the <appSettings> element in your App.config or Web.config file:

<appSettings>
    <add key="Insight.Token" value="bb61600f-f766-451e-b55f-9204f536a79f" />
</appSettings>

Here is an example of how to specify the credentials as Windows Azure role configuration settings:

<ServiceConfiguration serviceName="MyApp" osFamily="3" osVersion="*" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" schemaVersion="2013-03.2.0">
    <Role name="MyRole">
        <Instances count="2" />
        <ConfigurationSettings>
            <Setting name="Insight.Token" value="bb61600f-f766-451e-b55f-9204f536a79f" />
        </ConfigurationSettings>
    </Role&rt;
</ServiceConfiguration>

log4net Logging Context Information in Web Applications

In web application it is often helpful to use Log4net’s built-in ability to log additional contextual information with each log message. This works particularly well in combination with Log Management (InsightOps)’ log message indexer, which can identify any key-value-pairs in the incoming log message and index those for fast search and retrieval.

Here is an example of how additional web-specific contextual log information can be added to the layout of the Log Management (InsightOps) appender in a format that the Log Management (InsightOps) parser will recognize and index:

<appender name="InsightAppender" type="log4net.Appender. R7InsightAppender, R7Insight.Log4net">
    ...
    <layout type="log4net.Layout.PatternLayout">
        <param name="ConversionPattern" value="%d %logger %level% %m%nSessionId='%aspnet-request{ASP.NET_SessionId}'; Username='%aspnet-request{AUTH_USER}'; ClientIpAddress='%aspnet-request{REMOTE_ADDR}'; ClientUserAgent='%aspnet-request{HTTP_USER_AGENT}'; ServerName='%aspnet-request{SERVER_NAME}'; RequestMethod='%aspnet-request{REQUEST_METHOD}'; RequestUrl='%aspnet-request{URL}'; RequestQueryString='%aspnet-request{QUERY_STRING}'; RequestCookies='%aspnet-request{HTTP_COOKIE}';%n" />
    </layout>
</appender>

Token-Based Logging with log4net

Our recommended method of sending messages to Log Management (InsightOps) is via Token TCP over port 10000. To use this method, select Token TCP as the source type when creating a new log in the Log Management (InsightOps) UI, and then paste the token that is printed beside the log in the value for the Insight.Token credential setting.

Sending Log Data over SSL/TLS with log4net

The Log Management (InsightOps) appender supports sending log data over SSL/TLS with both of the above logging methods by setting the useSsl logging setting to true in the appender definition. This is more secure but may have a performance impact.

Sending Log Messages from Your Application with log4net

With installation and configuration out of the way, you are ready to send log data to Log Management (InsightOps).

In each class you wish to log from, add the following using directive at the top if it’s not already there:

using log4net;

Then create a logger object at the class level:

private static readonly ILog m_Logger = LogManager.GetLogger(typeof({YOURCLASSNAMEHERE}).FullName);

Be sure to enter the name of current class instead of {YOURCLASSNAMEHERE} above. This creates a logger with the same name as the current class, which organizes the log4net configuration hierarchy according to your code namespace hierarchy. This provides both clarity when reading the logs, and convenience when configuring different log levels for different areas of your code.

Now within your code in that class, you can log using log4net as normal and it will log to Rapid7 Insight.

Examples:

m_Logger.Debug("Debugging message");
m_Logger.Info("Informational message");
m_Logger.Warn("Warning message");
m_Logger.Error("Error message", ex);

Complete code example:

using log4net;

public class HomeController : Controller
{
    private static readonly ILog m_Logger = log4net.LogManager.GetLogger(typeof(HomeController).FullName);

    public ActionResult Index()
    {
        m_Logger.Debug("Home page viewed.");       
        ViewBag.Message = "Welcome to ASP.NET MVC!";       
        m_Logger.Warn("This is a warning message!");       
        return View();
    }
}

Troubleshooting log4net

By default the Log Management (InsightOps) appender logs its own debug messages to log4net’s internal logger. Checking these debug messages can help figuring out why logging to Log Management (InsightOps) does not work as expected.

To disable log4net internal debug messages, set the log4net.Internal.Debug setting in the <appSettings> section of your App.config or Web.config file to false:

<appSettings>
    <add key="log4net.Internal.Debug" value="false" />
</appSettings>

If you would like to keep log4net internal debugging enabled in general, but disable Insight logger debug messages specifically, then change the debug parameter inside the <appender> element to false instead:

<appender name="LeAppender" type="log4net.Appender.InsightAppender, R7Insight.Log4net">
    <debug value="false" />
    ...
</appender>

Ensure that you followed the section of this readme regarding your AssemblyInfo.cs file.

Shutting Down log4net’s Logger

The Log Management (InsightOps) appender keeps an internal queue of log messages and communicates with the Log Management (InsightOps) system using a background thread which continuously sends messages from this queue. Because of this, when an application is shutting down, it is possible that some log messages might still remain in the queue and will not have time to be sent to Log Management (InsightOps) before the application domain is shut down.

To work around this potential problem, consider adding the following code to your application, which will block for a moment to allow the Log Management (InsightOps) appender to finish logging all messages in the queue. The AreAllQueuesEmpty() blocks for a specified time and then returns true or false depending on whether the queues had time to become empty before the method returns.

{
    // This will give the background thread some time to finish sending messages to Log Management (InsightOps).
    var numWaits = 3;
    while (!R7InsightCore.Net.AsyncLogger.AreAllQueuesEmpty(TimeSpan.FromSeconds(5)) && numWaits > 0)
        numWaits--;
}