Scan Configuration Parameters

The scan configurations parameters are available in the AppSpider user interface or through the scan configuration file.

ScanConfig

ScanConfig is the top-level structure in the Scan Configuration File. ScanConfig's composite objects are presented in the Advanced Tab of the Scan Configuration Dialog. The scalar values of ScanConfig can only be modified by editing the ScanConfiguration file. After you manually edit the Scan Configuration File, AppSpider should be re-started.

ScanConfig object

The ScanConfig object contains the following objects:

• CrawlConfig - Defines the crawler parameters.
• AttackerConfig - Defines the attacker parameters.
• AttackPolicyConfig - Defines the attack policy. A list of attack modules for the scan and their parameters.
• AnalyzerConfig - Defines the analyzer parameters.
• AuthConfig - The authentication configuration. This structure contains everything related to authentication, login, re-login, logout detection.
• ProxyConfig - The proxy settings.
• RemediationConfig - The parameters for calculating remediation efforts.
• SSLCertConfig - SSL client certificate settings.
• NetworkSettingsConfig - The network parameters.
• PerformanceConfig - The performance parameter.
• SystemRecommendationsConfig - The parameters for computer hardware recommendations.
• HTTPHeadersConfig - The HTTP Headers.
• ManualCrawlingConfig - List of traffic log files to import.
• AutoSequenceConfig - Automatic sequence discovery settings.
• MacroConfig - List of macros for the scan.
• SeleniumConfig - List of selenium scripts and setting to run selenium scripts.
• WebServiceConfig - The web service configuration.
• ReportConfig - Report generation settings.
• WAFConfig - Deprecated
• ScheduleConfig - Deprecated
• SiteTechnologyConfig - The site technology settings.
• OneTimeTokenConfig - The parameters of One-Time Tokens (XSRF tokens).
• CVSSConfig - The CVSS configuration.
• ParameterParserConfig - The custom URL parameter parsers.
• ParameterValueConfig - The description of parameters used to populate form controls.

Detailed Logging

Detailed logging increases the amount of the information ScanEngine logs during scan execution. That extra information appears in the operation log. While it is useful to enable detailed logging for debugging, it is recommended to disable it for normal scan execution. Enabling detailed logging significantly increases the size of the log files on disk and slows the scan.

PauseOnRecoverableError

Some of the errors that can be corrected are:

• Re-login problem
• Out of disk space
• Out of memory

MaxTrafficFiles

The scanner removes old traffic files after number of traffic file reaches the number specified in this parameter. It follows First In, First Out (FIFO).

CrawlConfig

MaxDomain

Domain is a host name and protocol (https://www.live.com) in the URL. For instance, windows.microsoft.com and www.microsoft.com are different domains. If AppSpider finds a URL with an explicit IP address that matches a host name in another URL (for instance http://localhost/page.html and http://127.0.0.1/page.html), AppSpider will consider those as two different domains. Also, the protocol is a part of the domain name, and, as a result, https://localhost and http://localhost are two different domains.

MaxPerLinkCrawlResults

This option limits how many resources that have the same URL but different variations of POST parameters can be crawled.

MaxPerDirChildNodes

This parameter includes Crawl Results and sub-directories. The crawler will stop crawling new resources in a directory if one of the limits specified in MaxPerDirCrawlResult or MaxPerDirChildNodes is reached for that directory. This implies that the value of MaxPerDirCrawlResult should be greater or equal to the value of MaxPerDirChildNodes.

MaxBlackListExtCrawlResults

Maximum number of Web Resources that are on BlackListExtensionList and GrayListExtensionList that AppSpider is allowed to crawl. Even if the resource should be blacklisted based on extension, AppSpider will still crawl a small number of those resources the number of resources specified in this parameter.

MaxAttackFeedbackLinksCount

Crawler monitors traffic of attack modules and tries to find new links in the traffic. This parameter specified maximum number of new Web Resource found in the responses received by attack modules. Some attack requests result is responses that contain some of the attack payload or invalid links. For example, its very common for website to return a reference to a URL that could help user to solve the problem. That reference often include the description of the problem.

If the attack set the following request: GET /users/userinfo?userid=alert(‘111’)

The response to the attack could have the following statement and a new link.

Invalid parameter to the request /users/userinfo?userid=alert(‘111’): <a href=”/mgmt/reporterror.php=Invalid parameter URL: /users/userinfo?userid=alert(‘111’)”>Report error</a>

Every attack would result in new Web Resource /mgmt/reporterror.php with different parameter values. It could make sense to analyze that web resource once, it does not make to analyze all the web resources. AppSpider tries automatically detect those invalid links and avoid crawling them. The MaxAttackFeedbackLinksCount configuration parameter is a safety net in case invalid link detection algorithm fails.

MaxPerFileNameCrawlResults

During counting number of resources, only file name of the URL is considered and path is ignored). For example the following two URLs are considered to have the same File Name, because the filename is the same for both URLs.

• http://mysite/customers/get_address.php?id=23
• http://mysite/suppliers/get_address.php?id=78

RecursionDepth

This parameter defines how many times a part of a URL can be repeated. For example, if the recursion depth is set to 2, URL 1 will be crawled, while the URL 2 will be ignored because its recursion depth is 3.

1. www.site.com/dir1/dir2/dir1/dir2/file.txt
2. www.site.com/dir1/dir2/dir1/dir2/dir1/dir2/file.txt

MaxReportedLinks

The crawler almost always sees more links that it will crawl. Often it discovers more links that it will crawl. Many web resources are ignored because one or more crawler limitation. This parameter describes how many web resources will be stored in the database on top of the web resources that were be crawled. While web resources take significantly less space in the database then rrawl Results, they still take some space, so it is recommended to keep this number below the value specified in parameter MaxCrawlResults.

NotInsertedLinkCountThreshold

AppSpider reports ignored links in the User Log so that the user could easily notice that some URLs were unintentionally ignored and correct the problem by Only important messages should be reported in User Log. AppSpider ignores many links during crawling (for instance, out-of-domain links), on an average site the number of ignore links can be in thousands. To avoid cluttering User Log, it is recommended to keep this number low.

SaveReferences

Storing references significantly increases the size of the database, and it is advised not to enable this feature.

UseBrowser

This flag only affect using browser for crawling. It has no affect on using browser for macros, saequences or attacks.

ShowBrowser

This flag was designed to be used to debug various crawling problems. It is advised to disable this feature for regular scans. If this feature enabled, it is recommended to make the scan Single-Threaded. This way, only 1 browser window will be shown at any given moment.

RestrictToManualCrawling

If the value is set to 1, AppSpider will analyzed/attacked only the requests that it imported from the proxy logs. If imported responses contain other links, those links will not be crawler, analyzed or attacked. If the imported traffic has 5 requests, only 5 requests will be analyzed or attacked.

PageEqualThreshhold

PageEqualThreshhold deals with randomness in the responses. Some pages always have advertisement frames that website randomly inserts in the responses. This parameter allows AppSpider to ignore that random content.

The more similar the pages are, the higher is their similarity coefficient. Two responses with with similarity coefficient above the value of the parameter are considered by AppSpider to be identical. This information is used in several components in AppSpider, when comparing responses with custom 404 response or to determine whether that response is seen too often and can be ignored. It is recommended to not change the value of this parameter.

PageSimilarThreshhold

PageSimilarThreshhold deals with pages that return logically equivalent responses. For example, the shopping cart’s checkout page with a sweater in the shopping cart will look very similar to the page with a scarf in the shopping cart. This parameter helps AppSpider to understand what pages are similar. Several attacks and the analyzer use analysis for page similarity.

The more similar the pages are, the higher is their similarity coefficient. Two responses with with similarity coefficient above the value of the parameter are considered by AppSpider to be similar. It is recommended to not change the value of this parameter.

SearchForUrls

For example, this page HTML page with an HTML comment block:

html
1
<a href=”/admin/show_users.php”>Show Users</a><br>
2
<!-- 
3
  Do not forget that we need to remove ‘/admin/rebootserver.php’ when we are done debugging
4
--->
5
<a href=”/admin/server_info.php”>Server Information</a>

If this flag is set to 1, AppSpider will find URL /admin/rebootserver.php. If the flag is set to 0, AppSpider won’t find the URL.

MaxWebResourcesOverhead

Even if those extra resources are added to the queue, the crawler will stop crawling once it reached the crawled number of links specified in the MaxCrawlResult option.

AuthConfig

AssumeSuccessfulLogin

This parameter is often used in conjunction with macro login when the user can see in the browser that AppSpider logged in and does not want to craft a regular expression that detects a logged in state.

VerifyNotLoggedin

If during scanning a false positive logout was detected in a responses, AppSpider will try to re-login into session that was perfectly valid. This parameter configures how AppSpider behaves in this situations. If the value of the parameter is set to 1, AppSpider first check whether the user is already logged in, prior to starting login process. If it is set to 0, AppSpider will reset session .

AnalyzerConfig

AttackerConfig

MaxNormalizedSameNameParameterAttackPointsPerLink

For instance if a page has parameter params[1], params[2], params[3], all those parameters will have the same normalized name: params[].